Oct 30 04:41:14.600255 ip-10-0-139-228 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 04:41:14.600266 ip-10-0-139-228 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 04:41:14.600275 ip-10-0-139-228 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 04:41:14.600584 ip-10-0-139-228 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 04:41:24.237895 ip-10-0-139-228 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 04:41:24.237906 ip-10-0-139-228 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d601d26e98b94aeabe99552f9dc20885 -- Oct 30 04:43:04.953529 ip-10-0-139-228 systemd[1]: Starting Kubernetes Kubelet... Oct 30 04:43:05.417204 ip-10-0-139-228 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:05.417204 ip-10-0-139-228 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 04:43:05.417204 ip-10-0-139-228 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:05.417204 ip-10-0-139-228 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:05.423968 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423876 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.423968 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423964 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.423968 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423969 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.423968 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423973 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.423968 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423976 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423980 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423983 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423985 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423990 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423994 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.423997 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424000 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424003 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424005 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424008 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424011 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424013 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424016 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424019 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424021 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424024 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424026 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424029 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424035 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.424165 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424038 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424041 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424044 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424046 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424049 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424052 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424055 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424057 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424060 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424064 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424068 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424072 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424075 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424079 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424083 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424086 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424091 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424094 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424096 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.424632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424099 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424102 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424104 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424107 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424109 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424112 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424114 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424117 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424119 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424122 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424125 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424127 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424130 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424133 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424135 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424138 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424145 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424147 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424150 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424153 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.425076 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424155 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424158 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424160 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424163 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424166 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424169 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424171 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424175 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424177 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424180 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424182 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424185 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424187 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424190 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424192 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424195 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424198 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424200 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424203 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424206 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.425562 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424209 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424212 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424231 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424236 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424240 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424244 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424249 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424252 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424255 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424258 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424260 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424263 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424267 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424269 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424272 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424275 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424277 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424280 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424283 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424285 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.426042 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424288 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424291 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.424293 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425108 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425116 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425120 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425124 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425127 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425130 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425132 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425135 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425137 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425143 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425149 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425152 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425155 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425157 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425160 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425162 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.426632 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425165 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425167 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425170 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425173 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425176 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425178 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425181 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425183 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425187 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425190 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425193 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425196 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425198 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425201 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425203 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425206 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425208 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425211 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425214 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.427093 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425232 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425235 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425237 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425240 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425242 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425245 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425250 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425252 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425256 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425259 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425261 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425264 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425267 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425270 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425272 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425275 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425277 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425280 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425282 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425285 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.427556 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425287 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425291 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425294 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425296 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425299 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425301 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425304 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425307 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425309 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425312 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425314 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425317 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425319 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425322 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425324 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425327 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425330 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425332 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425336 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425338 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.428022 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425342 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425345 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425349 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425353 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425358 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425362 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425365 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425368 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425371 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425374 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425377 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425380 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425383 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425387 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425389 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425392 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425394 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425397 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425400 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.428526 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425402 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425405 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425407 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425410 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425412 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425415 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425417 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425419 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425422 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425425 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425427 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425431 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425957 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425964 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425968 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425971 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425973 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425976 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425978 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.428974 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425981 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425984 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425986 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425989 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425992 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425994 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.425997 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426000 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426003 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426005 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426008 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426010 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426013 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426016 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426019 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426021 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426024 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426026 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426029 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.429426 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426031 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426034 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426036 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426039 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426042 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426046 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426049 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426052 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426054 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426058 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426060 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426063 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426066 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426068 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426070 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426073 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426075 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426078 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426081 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426083 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.429885 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426086 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426090 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426094 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426097 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426100 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426102 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426105 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426107 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426110 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426112 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426115 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426118 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426121 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426123 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426126 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426129 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426131 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426135 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426138 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.430388 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426141 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426143 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426146 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426149 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426152 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426154 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426156 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426159 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426163 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426167 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426170 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426173 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426176 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426179 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426181 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426184 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426186 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426189 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426191 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426194 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.430827 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426196 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426199 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426201 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426204 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426207 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426210 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426212 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426227 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426230 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426232 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426236 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426239 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426241 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426244 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426247 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426249 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426252 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426255 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426258 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426260 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.431350 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.426263 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426362 2562 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426370 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426379 2562 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426386 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426393 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426397 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426404 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426409 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426412 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426415 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426419 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426422 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426426 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426428 2562 flags.go:64] FLAG: --cgroup-root="" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426431 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426434 2562 flags.go:64] FLAG: --client-ca-file="" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426438 2562 flags.go:64] FLAG: --cloud-provider="external" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426441 2562 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426447 2562 flags.go:64] FLAG: --cluster-domain="" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426450 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426453 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426458 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426461 2562 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 04:43:05.431819 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426470 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426473 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426476 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426479 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426482 2562 flags.go:64] FLAG: --contention-profiling="false" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426486 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426489 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426492 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426495 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426499 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426502 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426505 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426508 2562 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426513 2562 flags.go:64] FLAG: --enable-server="true" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426516 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426521 2562 flags.go:64] FLAG: --event-burst="100" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426524 2562 flags.go:64] FLAG: --event-qps="50" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426527 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426530 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426533 2562 flags.go:64] FLAG: --eviction-hard="" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426538 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426541 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426544 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426547 2562 flags.go:64] FLAG: --eviction-soft="" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426551 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 04:43:05.432388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426554 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426557 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426560 2562 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426563 2562 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426566 2562 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426570 2562 flags.go:64] FLAG: --feature-gates="" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426574 2562 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426577 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426580 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426583 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426587 2562 flags.go:64] FLAG: --healthz-port="10248" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426590 2562 flags.go:64] FLAG: --help="false" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426593 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426597 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426600 2562 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426603 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426607 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426610 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426613 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426616 2562 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426620 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426623 2562 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426626 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426629 2562 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 04:43:05.433007 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426632 2562 flags.go:64] FLAG: --kube-reserved="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426636 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426639 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426659 2562 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426662 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426666 2562 flags.go:64] FLAG: --lock-file="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426668 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426671 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426675 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426680 2562 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426683 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426686 2562 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426690 2562 flags.go:64] FLAG: --logging-format="text" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426693 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426697 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426699 2562 flags.go:64] FLAG: --manifest-url="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426702 2562 flags.go:64] FLAG: --manifest-url-header="" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426707 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426710 2562 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426713 2562 flags.go:64] FLAG: --max-pods="110" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426716 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426719 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426723 2562 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426726 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426729 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 04:43:05.433671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426732 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426735 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426742 2562 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426746 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426749 2562 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426753 2562 flags.go:64] FLAG: --pod-cidr="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426756 2562 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426758 2562 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426761 2562 flags.go:64] FLAG: --pods-per-core="0" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426764 2562 flags.go:64] FLAG: --port="10250" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426767 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426770 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0789f657a9600d716" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426773 2562 flags.go:64] FLAG: --qos-reserved="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426776 2562 flags.go:64] FLAG: --read-only-port="10255" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426779 2562 flags.go:64] FLAG: --register-node="true" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426782 2562 flags.go:64] FLAG: --register-with-taints="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426786 2562 flags.go:64] FLAG: --registry-burst="10" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426789 2562 flags.go:64] FLAG: --registry-qps="5" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426792 2562 flags.go:64] FLAG: --reserved-cpus="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426795 2562 flags.go:64] FLAG: --reserved-memory="" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426798 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426801 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426804 2562 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426807 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426810 2562 flags.go:64] FLAG: --runonce="false" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426813 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 04:43:05.434294 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426816 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426819 2562 flags.go:64] FLAG: --seccomp-default="false" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426822 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426825 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426828 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426831 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426834 2562 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426837 2562 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426840 2562 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426844 2562 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426847 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426850 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426853 2562 flags.go:64] FLAG: --system-cgroups="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426855 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426861 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426864 2562 flags.go:64] FLAG: --tls-cert-file="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426866 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426870 2562 flags.go:64] FLAG: --tls-min-version="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426874 2562 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426876 2562 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426879 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426882 2562 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426885 2562 flags.go:64] FLAG: --v="2" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426889 2562 flags.go:64] FLAG: --version="false" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426894 2562 flags.go:64] FLAG: --vmodule="" Oct 30 04:43:05.435800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426898 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.426902 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427071 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427076 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427079 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427082 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427085 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427088 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427091 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427094 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427096 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427099 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427103 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427105 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427108 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427110 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427114 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427117 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427120 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427122 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.436392 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427125 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427127 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427130 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427132 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427135 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427138 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427140 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427142 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427145 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427148 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427150 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427153 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427155 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427158 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427160 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427163 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427165 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427168 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427170 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.436860 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427173 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427175 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427178 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427180 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427183 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427185 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427188 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427192 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427195 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427199 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427202 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427204 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427207 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427209 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427212 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427226 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427229 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427232 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427235 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.437376 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427237 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427242 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427245 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427248 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427251 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427254 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427257 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427259 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427262 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427265 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427267 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427270 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427273 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427275 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427278 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427281 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427283 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427286 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427288 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427291 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.437820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427296 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427298 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427302 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427305 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427308 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427310 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427313 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427315 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427318 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427321 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427323 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427326 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427329 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427331 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427334 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427337 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427341 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427345 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427348 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427351 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.438302 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427354 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427357 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427360 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427362 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427365 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427367 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427370 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427373 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427375 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427378 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.427404 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427441 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427445 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427450 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427456 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427459 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.438771 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427461 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427465 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427469 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427472 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427475 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427478 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427481 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427483 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427486 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427488 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427491 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427494 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427497 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427499 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427502 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427505 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427508 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427510 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427513 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427516 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.439213 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427519 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427522 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427524 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427527 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427529 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427532 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427535 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427537 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427540 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427544 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427548 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427551 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427553 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427556 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427558 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427561 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427564 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427566 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427569 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427572 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.439702 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427574 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427577 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427580 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427582 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427585 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427587 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427590 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427592 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427595 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427597 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427600 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427602 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427605 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427608 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427610 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427612 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427615 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427617 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427620 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.440171 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427622 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427625 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427628 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427634 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427638 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427640 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427643 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427645 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427648 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427650 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427653 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427655 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427658 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427661 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427664 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427667 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427669 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427672 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427674 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.440631 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427677 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427679 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427682 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427684 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427687 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427689 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427692 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427694 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427697 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427699 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427702 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427705 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427707 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427709 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427712 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427718 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427721 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427724 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427727 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427730 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.441116 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427733 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427736 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.427738 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435109 2562 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435126 2562 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435141 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435146 2562 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435203 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435208 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435211 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435231 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435236 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435241 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435244 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435247 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435249 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435253 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435255 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435258 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435261 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.441643 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435263 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435266 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435270 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435275 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435278 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435281 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435283 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435286 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435289 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435291 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435294 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435297 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435300 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435302 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435305 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435307 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435312 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435316 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435318 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.442114 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435321 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435324 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435326 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435329 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435332 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435335 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435337 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435340 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435343 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435345 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435348 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435350 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435353 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435355 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435358 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435361 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435363 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435366 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435368 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435371 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.442604 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435373 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435376 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435378 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435381 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435383 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435386 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435389 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435391 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435393 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435397 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435401 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435403 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435406 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435409 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435412 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435414 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435417 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435419 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435422 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.443062 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435424 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435427 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435429 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435432 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435434 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435437 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435439 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435442 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435444 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435447 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435449 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435452 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435454 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435457 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435459 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435462 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435465 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435467 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435470 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.443652 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435473 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435475 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435478 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435482 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435484 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435487 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435491 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435496 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435499 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435502 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435504 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435507 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435509 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435512 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435515 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.444084 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435517 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435545 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435694 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435699 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435701 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435704 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435707 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435709 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435712 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435715 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435717 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435720 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435722 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435725 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435727 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435730 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.444567 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435732 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435735 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435737 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435740 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435742 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435745 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435748 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435751 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435753 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435756 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435758 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435761 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435764 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435766 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435769 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435771 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435774 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435776 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435779 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435781 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.444959 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435783 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435786 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435789 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435791 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435794 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435796 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435800 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435803 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435807 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435809 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435812 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435815 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435818 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435820 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435823 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435826 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435828 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435831 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435834 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.445467 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435836 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435839 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435842 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435845 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435847 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435850 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435852 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435855 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435857 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435860 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435862 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435865 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435867 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435870 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435872 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435875 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435878 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435880 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435883 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435885 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.445980 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435888 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435890 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435893 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435895 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435898 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435901 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435904 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435906 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435909 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435911 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435914 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435916 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435919 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435921 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435924 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435927 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435929 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435932 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435935 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.446571 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435937 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435939 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435942 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435945 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435948 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435952 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435955 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435957 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435960 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435962 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435965 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435967 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435970 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.435972 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.435995 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436026 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:05.447017 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436031 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436034 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436037 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436040 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436043 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436045 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436048 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436051 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436054 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436056 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436059 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436061 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436064 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436067 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436070 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436072 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436075 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436077 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436080 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:05.447404 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436082 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436085 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436088 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436093 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436096 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436098 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436101 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436103 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436106 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436109 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436111 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436114 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436116 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436119 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436121 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436124 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436127 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436129 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436132 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436134 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:05.447864 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436137 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436142 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436144 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436147 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436149 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436152 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436155 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436158 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436160 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436163 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436165 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436168 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436171 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436173 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436176 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436178 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436181 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436183 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436186 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436188 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:05.448351 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436191 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436193 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436196 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436198 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436201 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436203 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436206 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436209 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436212 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436228 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436232 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436235 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436238 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436243 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436246 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436249 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436252 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436254 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436257 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:05.448822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436260 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436263 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436266 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436269 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436271 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436274 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436276 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436279 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436282 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436284 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436287 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436289 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436292 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436295 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436297 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436300 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436302 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436305 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436308 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436310 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:05.449321 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436313 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436316 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436319 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436322 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436325 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436327 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:05.436331 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.436388 2562 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.438882 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.439999 2562 server.go:1014] "Starting client certificate rotation" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.440101 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.440911 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.442939 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:05.449781 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.447438 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:05.459622 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.459603 2562 log.go:25] "Validated CRI v1 runtime API" Oct 30 04:43:05.463599 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.463585 2562 log.go:25] "Validated CRI v1 image API" Oct 30 04:43:05.464434 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.464407 2562 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 04:43:05.467314 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.467287 2562 fs.go:135] Filesystem UUIDs: map[2656cefc-8d8a-434f-aa8a-8d786b3cd2b8:/dev/nvme0n1p4 3542657d-cc51-4bd7-ac7b-19cb06f630dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 04:43:05.467388 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.467314 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 04:43:05.472731 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.472608 2562 manager.go:217] Machine: {Timestamp:2026-10-30 04:43:05.471045316 +0000 UTC m=+0.377013376 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099562 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229f015a6493798740e583ba65ec2d SystemUUID:ec229f01-5a64-9379-8740-e583ba65ec2d BootID:d601d26e-98b9-4aea-be99-552f9dc20885 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:eb:36:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:eb:36:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:36:7c:4c:c4:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 04:43:05.472731 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.472727 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 04:43:05.472907 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.472877 2562 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 04:43:05.476678 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476650 2562 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 04:43:05.476813 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476679 2562 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 04:43:05.476895 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476815 2562 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 04:43:05.476895 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476823 2562 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 04:43:05.476895 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476869 2562 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:05.476895 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.476886 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:05.478151 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.478140 2562 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 04:43:05.482910 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.482898 2562 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 04:43:05.483031 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483021 2562 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 04:43:05.483842 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483832 2562 kubelet.go:493] "Attempting to sync node with API server" Oct 30 04:43:05.483880 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483845 2562 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 04:43:05.483880 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483855 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 04:43:05.483880 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483864 2562 kubelet.go:405] "Adding apiserver pod source" Oct 30 04:43:05.483880 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.483873 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 04:43:05.484276 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.484263 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:05.492645 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.492623 2562 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 04:43:05.501617 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.501604 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:05.502529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502517 2562 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 04:43:05.502578 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502547 2562 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 04:43:05.502727 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502717 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 04:43:05.502757 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502732 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 04:43:05.502757 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502739 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 04:43:05.502757 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502745 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 04:43:05.502757 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502750 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 04:43:05.502757 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.502756 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 04:43:05.503473 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503464 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 04:43:05.503473 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503473 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 04:43:05.503538 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503480 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 04:43:05.503538 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503486 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 04:43:05.503538 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503495 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 04:43:05.503538 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.503505 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 04:43:05.504907 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.504898 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 04:43:05.504907 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.504907 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 04:43:05.508726 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.508714 2562 server.go:1263] "Started kubelet" Oct 30 04:43:05.508901 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.508833 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 04:43:05.508971 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.508839 2562 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 04:43:05.508971 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.508930 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 04:43:05.509566 ip-10-0-139-228 systemd[1]: Started Kubernetes Kubelet. Oct 30 04:43:05.510693 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.510675 2562 server.go:317] "Adding debug handlers to kubelet server" Oct 30 04:43:05.510801 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.510748 2562 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 04:43:05.514825 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.514807 2562 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 04:43:05.517792 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.517774 2562 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 04:43:05.517792 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.517780 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 04:43:05.517935 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.517895 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:05.518613 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518596 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 04:43:05.518613 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518604 2562 factory.go:55] Registering systemd factory Oct 30 04:43:05.518752 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518622 2562 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 04:43:05.518752 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518638 2562 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 04:43:05.518752 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518625 2562 factory.go:223] Registration of the systemd container factory successfully Oct 30 04:43:05.518752 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518714 2562 reconstruct.go:97] "Volume reconstruction finished" Oct 30 04:43:05.518752 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518720 2562 reconciler.go:29] "Reconciler: start to sync state" Oct 30 04:43:05.518901 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518828 2562 factory.go:153] Registering CRI-O factory Oct 30 04:43:05.518901 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518845 2562 factory.go:223] Registration of the crio container factory successfully Oct 30 04:43:05.518901 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518899 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 04:43:05.519014 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518920 2562 factory.go:103] Registering Raw factory Oct 30 04:43:05.519014 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.518935 2562 manager.go:1199] Started watching for new ooms in manager Oct 30 04:43:05.520012 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.519998 2562 manager.go:319] Starting recovery of all containers Oct 30 04:43:05.528896 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.528824 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 04:43:05.530143 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.530115 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 04:43:05.530241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.530160 2562 manager.go:324] Recovery completed Oct 30 04:43:05.533619 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.530252 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-228.ec2.internal.18e335081862305b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-228.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-228.ec2.internal,},FirstTimestamp:2026-10-30 04:43:05.508687963 +0000 UTC m=+0.414656023,LastTimestamp:2026-10-30 04:43:05.508687963 +0000 UTC m=+0.414656023,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-228.ec2.internal,}" Oct 30 04:43:05.535698 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.535670 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.540465 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.540433 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.540532 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.540483 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.540532 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.540498 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.541092 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541079 2562 cpu_manager.go:226] "Starting" policy="none" Oct 30 04:43:05.541137 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541092 2562 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 04:43:05.541137 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541108 2562 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 04:43:05.541137 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541080 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.541829 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541813 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.541910 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541840 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.541910 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.541859 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.543419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.543402 2562 policy_none.go:50] "Start" Oct 30 04:43:05.543419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.543417 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 04:43:05.543549 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.543428 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 04:43:05.545459 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.545445 2562 policy_none.go:44] "Start" Oct 30 04:43:05.550414 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.550394 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlkjn" Oct 30 04:43:05.552448 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.552425 2562 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 04:43:05.552532 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.552470 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 04:43:05.552532 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.552508 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 04:43:05.557499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.557483 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlkjn" Oct 30 04:43:05.572963 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.572879 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-228.ec2.internal.18e335081a47121e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-228.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-228.ec2.internal,},FirstTimestamp:2026-10-30 04:43:05.540465182 +0000 UTC m=+0.446433252,LastTimestamp:2026-10-30 04:43:05.540465182 +0000 UTC m=+0.446433252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-228.ec2.internal,}" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.572986 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.586380 2562 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589249 2562 manager.go:341] "Starting Device Plugin manager" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.589280 2562 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589292 2562 server.go:87] "Starting device plugin registration server" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589541 2562 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589552 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589650 2562 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589713 2562 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.589719 2562 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.590282 2562 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.590329 2562 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.591073 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.592330 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.592353 2562 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.592372 2562 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 04:43:05.597555 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.592408 2562 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 04:43:05.606499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.606484 2562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:05.690625 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.690552 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.691515 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.691501 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.691583 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.691531 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.691583 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.691546 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.691583 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.691570 2562 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.692610 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.692580 2562 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal"] Oct 30 04:43:05.692726 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.692643 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.693392 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.693378 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.693448 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.693412 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.693448 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.693427 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.695669 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.695655 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.695806 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.695780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.695806 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.695809 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.696384 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696366 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.696460 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696372 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.696460 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696420 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.696460 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696432 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.696460 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696395 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.696641 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.696481 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.698710 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.698694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.698793 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.698723 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:05.699508 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.699495 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:05.699578 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.699518 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:05.699578 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.699527 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:05.709586 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.709568 2562 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.709663 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.709595 2562 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-228.ec2.internal\": node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:05.719872 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.719849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.719937 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.719878 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.719937 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.719893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.728146 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.728123 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-228.ec2.internal\" not found" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.735425 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.735408 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-228.ec2.internal\" not found" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.750072 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.750054 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:05.820012 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.819981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.820012 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.820011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.820238 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.820030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.820238 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.820062 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.820238 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.820074 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.820238 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:05.820095 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:05.850560 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.850528 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:05.950694 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:05.950619 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.033492 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.033466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:06.040818 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.040701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:06.050740 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.050720 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.151263 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.151209 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.252830 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.252747 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.353718 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.353690 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.440141 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.440102 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 04:43:06.440641 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.440292 2562 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 04:43:06.454817 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.454790 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.555492 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.555421 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.563474 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.563441 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:06 +0000 UTC" deadline="2028-04-02 22:08:12.5467322 +0000 UTC" Oct 30 04:43:06.563474 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.563473 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12497h25m5.983264769s" Oct 30 04:43:06.632998 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.632969 2562 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:06.654271 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.654242 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:06.655574 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.655553 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.657147 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.657130 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:06.657204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.657167 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:06.657204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.657179 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:06.741286 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:06.741241 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1c879f72343f788edee4e1a247e807.slice/crio-bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4 WatchSource:0}: Error finding container bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4: Status 404 returned error can't find the container with id bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4 Oct 30 04:43:06.741558 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:06.741536 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd885c44a562061a49f43990cf7668c0.slice/crio-9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb WatchSource:0}: Error finding container 9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb: Status 404 returned error can't find the container with id 9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb Oct 30 04:43:06.748823 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:06.748808 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:43:06.756190 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.756170 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.857073 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.856973 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:06.958169 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:06.958125 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:07.007968 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.007934 2562 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:07.037931 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.037908 2562 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:07.120972 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.120887 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:07.146382 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.146350 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:07.147702 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.147678 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:07.168992 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.168967 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:07.484469 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.484387 2562 apiserver.go:52] "Watching apiserver" Oct 30 04:43:07.502333 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.502300 2562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:07.518619 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.518593 2562 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 04:43:07.519028 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.519004 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal","openshift-multus/network-metrics-daemon-2fszz","openshift-ovn-kubernetes/ovnkube-node-s445b","kube-system/global-pull-secret-syncer-7tbzb","kube-system/konnectivity-agent-nrrc2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l","openshift-cluster-node-tuning-operator/tuned-s7nn8","openshift-multus/multus-8krtz","openshift-network-diagnostics/network-check-target-6v95n","openshift-dns/node-resolver-t9k5j","openshift-multus/multus-additional-cni-plugins-nfrbv","openshift-network-operator/iptables-alerter-577zz","openshift-image-registry/node-ca-lphjh"] Oct 30 04:43:07.520315 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.520286 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 04:43:07.521749 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.521725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.524133 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.524110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.526588 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.526566 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.526680 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.526663 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:07.529037 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.529016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.532779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.532756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.532869 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.532824 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:07.535384 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.535360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.538580 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.538560 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.543378 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.542844 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.544876 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.544845 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.544958 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.544949 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 04:43:07.545014 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.544966 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.545014 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.544969 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.545116 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545078 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 04:43:07.545116 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545105 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\"" Oct 30 04:43:07.545197 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545137 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 04:43:07.545197 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545166 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\"" Oct 30 04:43:07.545328 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545216 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.545328 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545259 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 04:43:07.545438 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.545418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.552170 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.548610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:07.552170 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.548696 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:07.553743 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.553306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.556047 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.556025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.600321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.600239 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" event={"ID":"cd885c44a562061a49f43990cf7668c0","Type":"ContainerStarted","Data":"9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb"} Oct 30 04:43:07.601826 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.601802 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerStarted","Data":"bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4"} Oct 30 04:43:07.619873 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.619850 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 04:43:07.629193 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.629331 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629331 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.629331 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.629511 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.629511 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629511 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.629511 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.629511 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zk2\" (UniqueName: \"kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpll\" (UniqueName: \"kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.629767 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9z2\" (UniqueName: \"kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjck\" (UniqueName: \"kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629875 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjz5\" (UniqueName: \"kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhkh\" (UniqueName: \"kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.629989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.630234 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.631035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmdh\" (UniqueName: \"kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.630987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.631889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464tl\" (UniqueName: \"kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.632706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.631550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.653831 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653793 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\"" Oct 30 04:43:07.653831 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653793 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653886 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653915 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653933 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.653956 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654004 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.654059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654018 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654078 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654167 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654209 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654217 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654237 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654256 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654264 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 04:43:07.654463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654305 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\"" Oct 30 04:43:07.654846 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654815 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.654900 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654888 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\"" Oct 30 04:43:07.654954 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654912 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 04:43:07.655033 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.655022 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.655268 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.655223 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 04:43:07.655397 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.654309 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 04:43:07.655836 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.655612 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 04:43:07.655836 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.655681 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\"" Oct 30 04:43:07.655975 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.655882 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\"" Oct 30 04:43:07.656529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.656094 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 04:43:07.666479 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.666453 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 04:43:07.666590 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.666538 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5k2hw" Oct 30 04:43:07.732546 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.732686 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732708 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.732737 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.732777 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:08.232752241 +0000 UTC m=+3.138720318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732808 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732888 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732942 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732849 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732775 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.733062 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:07.733204 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733085 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.732989 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.733113 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:08.233095744 +0000 UTC m=+3.139063793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733146 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733205 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733223 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733657 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmdh\" (UniqueName: \"kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.733996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734090 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464tl\" (UniqueName: \"kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734538 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.734915 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734593 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734623 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734847 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734886 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.735595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734909 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734981 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735028 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zk2\" (UniqueName: \"kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735080 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735119 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735138 2562 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735163 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734501 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735210 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.734951 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.736419 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735229 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735325 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735327 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpll\" (UniqueName: \"kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735161 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735454 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735498 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735554 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735599 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735654 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735794 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.737321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735864 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735903 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735949 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.735986 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736024 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9z2\" (UniqueName: \"kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjck\" (UniqueName: \"kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjz5\" (UniqueName: \"kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736466 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736479 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhkh\" (UniqueName: \"kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736837 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.738176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736837 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.736918 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737099 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737392 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737437 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737455 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737513 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737614 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737649 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737659 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737688 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737705 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737744 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.739138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737888 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.740067 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.737963 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.740067 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.738016 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.740686 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.740662 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.740880 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.740744 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.740997 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.740932 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.742363 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.742339 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.756590 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.756561 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:07.756590 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.756590 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:07.756770 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.756604 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:07.756770 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:07.756704 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:08.256683257 +0000 UTC m=+3.162651322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:07.758258 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.758229 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmdh\" (UniqueName: \"kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.760968 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.760804 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zk2\" (UniqueName: \"kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.760968 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.760922 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464tl\" (UniqueName: \"kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:07.761789 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.761162 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhkh\" (UniqueName: \"kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.761789 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.761474 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9z2\" (UniqueName: \"kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.762950 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.762851 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.763144 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.763123 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpll\" (UniqueName: \"kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.763647 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.763624 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjz5\" (UniqueName: \"kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.770157 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.770133 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjck\" (UniqueName: \"kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.842548 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.842496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:07.850350 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.850319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:07.855767 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.855735 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3765b6_e817_4d17_8397_197521d8c888.slice/crio-4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7 WatchSource:0}: Error finding container 4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7: Status 404 returned error can't find the container with id 4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7 Oct 30 04:43:07.862568 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.862535 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5faa68a_1aba_434f_94e8_83bdeedb2f28.slice/crio-3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b WatchSource:0}: Error finding container 3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b: Status 404 returned error can't find the container with id 3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b Oct 30 04:43:07.875442 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.875420 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:07.875981 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.875928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:07.880788 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.880764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8krtz" Oct 30 04:43:07.889066 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.889044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:07.891362 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.891334 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0743f86_5e12_4a1d_baf8_fb1973fe4034.slice/crio-b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e WatchSource:0}: Error finding container b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e: Status 404 returned error can't find the container with id b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e Oct 30 04:43:07.892685 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.892655 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b32ff5_8157_470f_94f6_ce77e4bf3898.slice/crio-ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b WatchSource:0}: Error finding container ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b: Status 404 returned error can't find the container with id ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b Oct 30 04:43:07.896565 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.896534 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d95e075_6720_45ab_ab71_f8b5bc189697.slice/crio-7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a WatchSource:0}: Error finding container 7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a: Status 404 returned error can't find the container with id 7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a Oct 30 04:43:07.896650 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.896625 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:07.904620 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.904593 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaf9947_5532_47ce_aea5_dcb456c37684.slice/crio-139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f WatchSource:0}: Error finding container 139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f: Status 404 returned error can't find the container with id 139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f Oct 30 04:43:07.906103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.906084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:07.915125 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.915095 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d022eb_a646_4f3b_8037_4f992cab9c21.slice/crio-ebc044911c4ff44a3958ef383e104cb3f1d147c9857bfc9e3cf61ce5eff50de4 WatchSource:0}: Error finding container ebc044911c4ff44a3958ef383e104cb3f1d147c9857bfc9e3cf61ce5eff50de4: Status 404 returned error can't find the container with id ebc044911c4ff44a3958ef383e104cb3f1d147c9857bfc9e3cf61ce5eff50de4 Oct 30 04:43:07.922023 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.921997 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37948e1b_b416_4df5_9c56_7ff2e7996e48.slice/crio-18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3 WatchSource:0}: Error finding container 18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3: Status 404 returned error can't find the container with id 18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3 Oct 30 04:43:07.926371 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:07.926347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:07.938648 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:07.938621 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd538562_960c_4a89_843b_fc901b120aa6.slice/crio-947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c WatchSource:0}: Error finding container 947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c: Status 404 returned error can't find the container with id 947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c Oct 30 04:43:08.240779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.240645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:08.240779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.240703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:08.241028 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.240835 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:08.241028 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.240928 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:09.240899838 +0000 UTC m=+4.146867887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:08.241028 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.240836 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:08.241028 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.240987 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:09.240974363 +0000 UTC m=+4.146942430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:08.342330 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.341595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:08.342330 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.341829 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:08.342330 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.341852 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:08.342330 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.341865 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:08.342330 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.341941 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:09.341920743 +0000 UTC m=+4.247888813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:08.593177 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.592977 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:08.593177 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:08.593111 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:08.610547 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.610465 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9k5j" event={"ID":"37948e1b-b416-4df5-9c56-7ff2e7996e48","Type":"ContainerStarted","Data":"18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3"} Oct 30 04:43:08.627499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.627456 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b"} Oct 30 04:43:08.639355 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.639254 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nrrc2" event={"ID":"e0743f86-5e12-4a1d-baf8-fb1973fe4034","Type":"ContainerStarted","Data":"b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e"} Oct 30 04:43:08.653658 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.653616 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-577zz" event={"ID":"f5faa68a-1aba-434f-94e8-83bdeedb2f28","Type":"ContainerStarted","Data":"3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b"} Oct 30 04:43:08.656540 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.656445 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" event={"ID":"9d3765b6-e817-4d17-8397-197521d8c888","Type":"ContainerStarted","Data":"4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7"} Oct 30 04:43:08.661710 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.661679 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerStarted","Data":"947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c"} Oct 30 04:43:08.665678 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.665584 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" event={"ID":"a0d022eb-a646-4f3b-8037-4f992cab9c21","Type":"ContainerStarted","Data":"ebc044911c4ff44a3958ef383e104cb3f1d147c9857bfc9e3cf61ce5eff50de4"} Oct 30 04:43:08.670091 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.669424 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:08 +0000 UTC" deadline="2028-07-13 08:00:50.822802536 +0000 UTC" Oct 30 04:43:08.670091 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.669490 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14931h17m42.153316242s" Oct 30 04:43:08.672405 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.672348 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lphjh" event={"ID":"bbaf9947-5532-47ce-aea5-dcb456c37684","Type":"ContainerStarted","Data":"139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f"} Oct 30 04:43:08.680440 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:08.680412 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8krtz" event={"ID":"3d95e075-6720-45ab-ab71-f8b5bc189697","Type":"ContainerStarted","Data":"7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a"} Oct 30 04:43:09.252215 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.252070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:09.252215 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.252115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:09.252215 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.252216 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:09.252511 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.252288 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:11.252268745 +0000 UTC m=+6.158236817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:09.252511 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.252339 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:09.252511 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.252363 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:11.252355782 +0000 UTC m=+6.158323830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:09.353436 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.352745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:09.353436 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.352964 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:09.353436 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.353043 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:09.353436 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.353056 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:09.353436 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.353104 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:11.353090485 +0000 UTC m=+6.259058549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:09.595429 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.594443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:09.595429 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.594584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:09.595429 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.595217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:09.595429 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:09.595372 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:09.670954 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.670606 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:08 +0000 UTC" deadline="2028-05-20 01:50:22.717716479 +0000 UTC" Oct 30 04:43:09.670954 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.670900 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13629h7m13.046822874s" Oct 30 04:43:09.711359 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:09.711315 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" event={"ID":"cd885c44a562061a49f43990cf7668c0","Type":"ContainerStarted","Data":"0f2424c780cf6e8c5ad40749891457a85ed34a79f127990b247d3bafd2ee3364"} Oct 30 04:43:10.593279 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:10.593201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:10.593467 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:10.593353 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:10.724138 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:10.724085 2562 generic.go:356] "Generic (PLEG): container finished" podID="af1c879f72343f788edee4e1a247e807" containerID="bd163a301784c713afa0ece382b5c208a2b379efffab2584a9cc0b33ae99ef6f" exitCode=0 Oct 30 04:43:10.724635 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:10.724147 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerDied","Data":"bd163a301784c713afa0ece382b5c208a2b379efffab2584a9cc0b33ae99ef6f"} Oct 30 04:43:10.767500 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:10.767132 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" podStartSLOduration=3.767111208 podStartE2EDuration="3.767111208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:08.734145993 +0000 UTC m=+3.640114068" watchObservedRunningTime="2026-10-30 04:43:10.767111208 +0000 UTC m=+5.673079282" Oct 30 04:43:11.269289 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.269225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:11.269491 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.269300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:11.269491 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.269461 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:11.269613 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.269527 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:15.269506359 +0000 UTC m=+10.175474411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:11.269987 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.269966 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:11.270074 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.270028 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:15.270012125 +0000 UTC m=+10.175980175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:11.370595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.370560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:11.370753 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.370690 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:11.370753 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.370707 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:11.370753 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.370717 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:11.370855 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.370784 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:15.370771259 +0000 UTC m=+10.276739321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:11.595592 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.595512 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:11.595771 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.595653 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:11.596251 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.596064 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:11.596251 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:11.596158 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:11.733654 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.733542 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerStarted","Data":"39bf65cf40e2e38e5c9cb675e3b51f7354d6963d95907304469beb3f655a7a22"} Oct 30 04:43:11.740327 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.740291 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="57326eda45dc90a280878c75d4b91d7f8a989023ad6dce691e1baa756a2fee1f" exitCode=0 Oct 30 04:43:11.740496 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.740352 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"57326eda45dc90a280878c75d4b91d7f8a989023ad6dce691e1baa756a2fee1f"} Oct 30 04:43:11.817931 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:11.817782 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" podStartSLOduration=4.817760154 podStartE2EDuration="4.817760154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:11.774975855 +0000 UTC m=+6.680943927" watchObservedRunningTime="2026-10-30 04:43:11.817760154 +0000 UTC m=+6.723728227" Oct 30 04:43:12.592886 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:12.592803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:12.593070 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:12.592947 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:12.746227 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:12.746112 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nrrc2" event={"ID":"e0743f86-5e12-4a1d-baf8-fb1973fe4034","Type":"ContainerStarted","Data":"123addb194d094bad4e5d366d8545ed07eee004436345be6bf201459656a1ee6"} Oct 30 04:43:13.595932 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:13.595852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:13.596140 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:13.595995 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:13.596447 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:13.596426 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:13.596560 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:13.596540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:13.751650 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:13.751613 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lphjh" event={"ID":"bbaf9947-5532-47ce-aea5-dcb456c37684","Type":"ContainerStarted","Data":"28812c265f733c572968f746c2d945ad34a922a7e95d50c9a371af76dfa553ad"} Oct 30 04:43:13.790842 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:13.790765 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nrrc2" podStartSLOduration=3.554704055 podStartE2EDuration="7.790743228s" totalImagesPullingTime="4.236039173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:12.787435208 +0000 UTC m=+7.693403284" watchObservedRunningTime="2026-10-30 04:43:13.790743228 +0000 UTC m=+8.696711301" Oct 30 04:43:14.292955 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:14.292861 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:14.293529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:14.293480 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:14.334078 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:14.333266 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lphjh" podStartSLOduration=3.050826996 podStartE2EDuration="8.333242741s" totalImagesPullingTime="5.282415745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:13.791809762 +0000 UTC m=+8.697777838" watchObservedRunningTime="2026-10-30 04:43:14.333242741 +0000 UTC m=+9.239210818" Oct 30 04:43:14.592910 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:14.592816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:14.593102 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:14.592965 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:14.756800 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:14.756765 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" event={"ID":"9d3765b6-e817-4d17-8397-197521d8c888","Type":"ContainerStarted","Data":"5934aedd13bcfac2d1654c3fa2823c4e7659d20116602acec2b1e00ff6e69ba9"} Oct 30 04:43:15.305701 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.305653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:15.305883 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.305717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:15.305949 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.305897 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:15.306001 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.305965 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.305944743 +0000 UTC m=+18.211912797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:15.306498 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.306396 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:15.306498 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.306455 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.306439179 +0000 UTC m=+18.212407232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:15.406170 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.406123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:15.406406 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.406383 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:15.406515 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.406411 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:15.406515 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.406426 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:15.406515 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.406496 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.406476803 +0000 UTC m=+18.312444881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:15.594510 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.594480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:15.594660 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.594637 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:15.596535 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.595293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:15.596535 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:15.595435 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:15.742038 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.741999 2562 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 04:43:15.764889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.764323 2562 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:15.764889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:15.764810 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" event={"ID":"9d3765b6-e817-4d17-8397-197521d8c888","Type":"ContainerStarted","Data":"283f76c3bfb4a09bcbab15bf96ab676d1c259d4b0962ddc4696febf968652138"} Oct 30 04:43:16.592950 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.592715 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:16.593115 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:16.593089 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:16.602971 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.602857 2562 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:43:15.742027259Z","UUID":"fbcb82b6-c9e2-4024-8ec5-e22c89f81f95","Handler":null,"Name":"","Endpoint":""} Oct 30 04:43:16.606646 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.605724 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 04:43:16.606646 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.605767 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 04:43:16.772324 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.771658 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8krtz" event={"ID":"3d95e075-6720-45ab-ab71-f8b5bc189697","Type":"ContainerStarted","Data":"211eaad46a277c1f9271bb94a3c29678ad6feb304d161f6f80919a3647eba043"} Oct 30 04:43:16.774310 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.774263 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9k5j" event={"ID":"37948e1b-b416-4df5-9c56-7ff2e7996e48","Type":"ContainerStarted","Data":"9ef8202e20129b62f85ffc8b3a43db41c70addf30f1d7442561c7b024f35f71a"} Oct 30 04:43:16.777698 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.777671 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerStarted","Data":"9850f490eb3c957e3ef9734397ae7c0bdc65b305c8202a346909aa4ecd4059be"} Oct 30 04:43:16.814896 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.814840 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8krtz" podStartSLOduration=2.924518528 podStartE2EDuration="10.814821594s" totalImagesPullingTime="7.890303066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:16.814145707 +0000 UTC m=+11.720113793" watchObservedRunningTime="2026-10-30 04:43:16.814821594 +0000 UTC m=+11.720789708" Oct 30 04:43:16.858019 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:16.857955 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t9k5j" podStartSLOduration=2.542026011 podStartE2EDuration="10.857937689s" totalImagesPullingTime="8.315911678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:16.85697194 +0000 UTC m=+11.762940020" watchObservedRunningTime="2026-10-30 04:43:16.857937689 +0000 UTC m=+11.763905765" Oct 30 04:43:17.593621 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.593544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:17.593808 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:17.593671 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:17.594094 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.594075 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:17.594241 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:17.594197 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:17.782974 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.782938 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" event={"ID":"9d3765b6-e817-4d17-8397-197521d8c888","Type":"ContainerStarted","Data":"f85c2f7b10abc0ebbec95f339fd07d333220f420c931fdbe7d9033278a7edef3"} Oct 30 04:43:17.785337 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.785311 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="9850f490eb3c957e3ef9734397ae7c0bdc65b305c8202a346909aa4ecd4059be" exitCode=0 Oct 30 04:43:17.785516 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.785397 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"9850f490eb3c957e3ef9734397ae7c0bdc65b305c8202a346909aa4ecd4059be"} Oct 30 04:43:17.822666 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:17.822612 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" podStartSLOduration=3.178557765 podStartE2EDuration="11.822591326s" totalImagesPullingTime="8.644033561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:17.821205246 +0000 UTC m=+12.727173316" watchObservedRunningTime="2026-10-30 04:43:17.822591326 +0000 UTC m=+12.728559398" Oct 30 04:43:18.593131 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:18.593093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:18.593312 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:18.593229 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:18.789118 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:18.789088 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-577zz" event={"ID":"f5faa68a-1aba-434f-94e8-83bdeedb2f28","Type":"ContainerStarted","Data":"a157110402b9d099cecb9fe5cf5bbcafef99da1eefb5319ecff7f28c942c60f9"} Oct 30 04:43:18.826321 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:18.826277 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-577zz" podStartSLOduration=4.139070156 podStartE2EDuration="12.826261768s" totalImagesPullingTime="8.687191612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:18.826258131 +0000 UTC m=+13.732226204" watchObservedRunningTime="2026-10-30 04:43:18.826261768 +0000 UTC m=+13.732229838" Oct 30 04:43:19.592759 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.592720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:19.592948 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.592720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:19.592948 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:19.592847 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:19.592948 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:19.592939 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:19.793372 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.793338 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="1f75fca03ac329e5348cf4aefac2187054f5a9f753e2e1845a01a6525267c26b" exitCode=0 Oct 30 04:43:19.794071 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.793414 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"1f75fca03ac329e5348cf4aefac2187054f5a9f753e2e1845a01a6525267c26b"} Oct 30 04:43:19.795617 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.795589 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" event={"ID":"a0d022eb-a646-4f3b-8037-4f992cab9c21","Type":"ContainerStarted","Data":"160537bb78210e02996570d8861357e6a248f3635e849100c77e370c26bba51a"} Oct 30 04:43:19.906974 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:19.906879 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" podStartSLOduration=3.04156806 podStartE2EDuration="13.906865226s" totalImagesPullingTime="10.865297166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:19.906521804 +0000 UTC m=+14.812489874" watchObservedRunningTime="2026-10-30 04:43:19.906865226 +0000 UTC m=+14.812833295" Oct 30 04:43:20.593064 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:20.593032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:20.593252 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:20.593163 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:21.593621 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:21.593368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:21.594185 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:21.593722 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:21.594185 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:21.593452 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:21.594313 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:21.594254 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:21.803209 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:21.803170 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="0c31af2eb69b3a5c007f557b6deae1925acf0845e9700492a7f273fce3b275e4" exitCode=0 Oct 30 04:43:21.803364 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:21.803242 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"0c31af2eb69b3a5c007f557b6deae1925acf0845e9700492a7f273fce3b275e4"} Oct 30 04:43:22.035795 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:22.035720 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:22.035934 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:22.035873 2562 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:22.036697 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:22.036669 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:22.592792 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:22.592759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:22.592968 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:22.592865 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:23.369437 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.369348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:23.369437 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.369387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:23.369903 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.369496 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:23.369903 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.369520 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:23.369903 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.369553 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.369539009 +0000 UTC m=+34.275507056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:23.369903 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.369585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret podName:bcf81c88-4fb2-4655-bd29-f2114fad2d3a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.369567631 +0000 UTC m=+34.275535693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret") pod "global-pull-secret-syncer-7tbzb" (UID: "bcf81c88-4fb2-4655-bd29-f2114fad2d3a") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:23.469992 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.469959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:23.470138 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.470114 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:23.470284 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.470144 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:23.470284 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.470156 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hk9zw for pod openshift-network-diagnostics/network-check-target-6v95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:23.470284 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.470213 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw podName:ce27bdc4-89f2-4eed-a76d-57280fcb0daa nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.470195477 +0000 UTC m=+34.376163541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9zw" (UniqueName: "kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw") pod "network-check-target-6v95n" (UID: "ce27bdc4-89f2-4eed-a76d-57280fcb0daa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:23.592807 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.592782 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:23.592979 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.592949 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:23.593036 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.592983 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:23.593085 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:23.593056 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:23.810309 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810205 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"5c0135b430f13396fa6e7c41d346b9e9aacb09a20191b1fb9147734b1161940a"} Oct 30 04:43:23.810309 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810293 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"6724fde9f8437ebdab4db5ba9911921432c2ab5e44069503751f66125da56bd5"} Oct 30 04:43:23.810499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810335 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"a934b11cb21e4292f61f049dc91dfb188e2d7ea961349e4586c1753259f184d2"} Oct 30 04:43:23.810499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810365 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"f07f3d2b29fe3601b57e7086ac80214075a759f6908339956762d668f25fb509"} Oct 30 04:43:23.810499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810394 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"6f51c4ea7997526d88df13afccf080ddd356b750d6f515899b48d7d3d0e1d204"} Oct 30 04:43:23.810499 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.810427 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"150c41372940730a572c165a66a83681c8ba36b154f3050c3790cc84062f1118"} Oct 30 04:43:23.812869 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.812843 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="910476787ae2184d18bdbe760750a919940af30a12ff85657888e764ad5a818b" exitCode=0 Oct 30 04:43:23.812971 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:23.812885 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"910476787ae2184d18bdbe760750a919940af30a12ff85657888e764ad5a818b"} Oct 30 04:43:24.593279 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:24.593234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:24.593737 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:24.593351 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:24.817350 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:24.817315 2562 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="eaa435e9d13829d61da1b201fffaf1bc938224a41b2615527f1115f511c694fc" exitCode=0 Oct 30 04:43:24.817350 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:24.817353 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerDied","Data":"eaa435e9d13829d61da1b201fffaf1bc938224a41b2615527f1115f511c694fc"} Oct 30 04:43:25.593476 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:25.593391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:25.593866 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:25.593504 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:25.593866 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:25.593541 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:25.593866 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:25.593668 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:25.822596 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:25.822553 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" event={"ID":"dd538562-960c-4a89-843b-fc901b120aa6","Type":"ContainerStarted","Data":"d499a0f0855eabe2a15e985f153fe6bf81b634635ce30bb40831b50e51538fb7"} Oct 30 04:43:25.825391 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:25.825365 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"2b33c9ebb3eca1faf0e2bcf81d9718e9a5c98cbccdb58412101d746f337d1793"} Oct 30 04:43:25.888782 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:25.888695 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nfrbv" podStartSLOduration=7.951730921 podStartE2EDuration="19.888681171s" totalImagesPullingTime="11.93695025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:25.876264302 +0000 UTC m=+20.782232379" watchObservedRunningTime="2026-10-30 04:43:25.888681171 +0000 UTC m=+20.794649241" Oct 30 04:43:26.592562 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:26.592526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:26.592714 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:26.592625 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:27.593183 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.593105 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:27.593183 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.593132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:27.594070 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:27.593230 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:27.594070 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:27.593323 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:27.832999 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.832818 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" event={"ID":"e4b32ff5-8157-470f-94f6-ce77e4bf3898","Type":"ContainerStarted","Data":"0f47c5532e8b116136e4d4e371c00c3c0331e2ea7775ccd933366b66046b37f3"} Oct 30 04:43:27.833181 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.833048 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:27.833181 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.833094 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:27.848430 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.848350 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:27.913991 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:27.913945 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" podStartSLOduration=6.949052395 podStartE2EDuration="21.913933581s" totalImagesPullingTime="14.964881186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:27.913747145 +0000 UTC m=+22.819715436" watchObservedRunningTime="2026-10-30 04:43:27.913933581 +0000 UTC m=+22.819901688" Oct 30 04:43:28.592625 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:28.592581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:28.592796 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:28.592708 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:28.835080 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:28.835032 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:28.849398 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:28.849333 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:29.443570 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.443541 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tbzb"] Oct 30 04:43:29.443791 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.443674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:29.443791 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:29.443761 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:29.446470 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.446444 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6v95n"] Oct 30 04:43:29.446592 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.446558 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:29.446677 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:29.446654 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:29.447094 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.447068 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fszz"] Oct 30 04:43:29.447196 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:29.447183 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:29.447341 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:29.447320 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:31.593610 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:31.593579 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:31.593957 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:31.593589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:31.593957 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:31.593697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:31.593957 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:31.593699 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:31.593957 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:31.593802 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:31.593957 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:31.593887 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:33.593328 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:33.593293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:33.593777 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:33.593329 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:33.593777 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:33.593395 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v95n" podUID="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" Oct 30 04:43:33.593777 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:33.593488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:33.593777 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:33.593602 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fszz" podUID="5771f103-055c-43b2-b344-fb3f64cc0998" Oct 30 04:43:33.593777 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:33.593693 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tbzb" podUID="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" Oct 30 04:43:35.373039 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.373010 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeReady" Oct 30 04:43:35.373425 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.373149 2562 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 04:43:35.414291 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.414266 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j2vn"] Oct 30 04:43:35.417828 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.417809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:35.418291 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.418268 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc"] Oct 30 04:43:35.422228 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.422199 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9ff65"] Oct 30 04:43:35.422310 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.422291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.425981 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.425964 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cddr7"] Oct 30 04:43:35.426126 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.426110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:35.429721 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.429701 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:43:35.429963 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.429940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:35.434468 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.434452 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94"] Oct 30 04:43:35.434574 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.434559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.440541 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.440510 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q"] Oct 30 04:43:35.440790 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.440772 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.441052 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.441026 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.441328 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.441309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.443241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.442601 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 04:43:35.447649 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.447629 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-d82h8"] Oct 30 04:43:35.447728 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.447720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.450748 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.450727 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5"] Oct 30 04:43:35.451581 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.450839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.456585 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.455156 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:43:35.456585 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.456150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:35.459380 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.459361 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv"] Oct 30 04:43:35.459515 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.459496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.462635 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.462617 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74b99df667-5m6dw"] Oct 30 04:43:35.463565 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.462759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.467600 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.467554 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-84nvd"] Oct 30 04:43:35.468052 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.468019 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.471771 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.471753 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47"] Oct 30 04:43:35.471891 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.471873 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.474761 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.474741 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wf2dd"] Oct 30 04:43:35.474868 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.474853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:35.477599 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.477584 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-79pwp"] Oct 30 04:43:35.477703 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.477688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.481779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.480780 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hq5jn"] Oct 30 04:43:35.481779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.480923 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.487706 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487685 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j2vn"] Oct 30 04:43:35.487879 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487736 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9ff65"] Oct 30 04:43:35.487879 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487763 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cddr7"] Oct 30 04:43:35.487879 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487800 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc"] Oct 30 04:43:35.487879 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487822 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:43:35.487879 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487840 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.488132 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.487844 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94"] Oct 30 04:43:35.488132 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488110 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488139 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488154 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-d82h8"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488169 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74b99df667-5m6dw"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488189 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488210 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-79pwp"] Oct 30 04:43:35.488242 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488245 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-84nvd"] Oct 30 04:43:35.488407 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488259 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5"] Oct 30 04:43:35.488407 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488271 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:43:35.488643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.488632 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wf2dd"] Oct 30 04:43:35.489668 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.489647 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hq5jn"] Oct 30 04:43:35.535739 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.535707 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.535846 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.535802 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.535944 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.535960 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536000 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536036 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536047 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536060 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536077 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536085 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 04:43:35.536107 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536111 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536112 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536123 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.535960 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536037 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536086 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536193 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536274 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536283 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536302 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536319 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536326 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536447 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 04:43:35.536529 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.536465 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 04:43:35.537049 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.537033 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.537128 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.537080 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.537207 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.537192 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.537331 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.537318 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:35.549273 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549248 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 04:43:35.549360 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549327 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 04:43:35.549360 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549346 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 04:43:35.549485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549377 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 04:43:35.549485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549392 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 04:43:35.549485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549403 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 04:43:35.549485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549428 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 04:43:35.549485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549407 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549549 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549547 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9lcst\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549592 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549606 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549606 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmmt4\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549623 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 04:43:35.549699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549643 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 04:43:35.549990 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549725 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 04:43:35.549990 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549733 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 04:43:35.549990 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.549776 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\"" Oct 30 04:43:35.553148 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.553195 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.553195 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhsx\" (UniqueName: \"kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.553292 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzpt\" (UniqueName: \"kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt\") pod \"volume-data-source-validator-58c9f5f46d-jxfs5\" (UID: \"e6975d49-682f-42d6-9ff2-21b90112b948\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:35.553292 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.553378 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq48z\" (UniqueName: \"kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z\") pod \"kas-connection-checker-685895c7c9-cddr7\" (UID: \"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92\") " pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:35.553441 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.553441 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.553509 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.553509 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.553562 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.553562 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.553562 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbhw\" (UniqueName: \"kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.553705 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvhq\" (UniqueName: \"kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.553705 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.553705 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.553705 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj76p\" (UniqueName: \"kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.553860 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.553945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.554103 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l2f\" (UniqueName: \"kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.554332 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8ms\" (UniqueName: \"kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms\") pod \"kas-connection-checker-685895c7c9-9ff65\" (UID: \"93773016-1fde-4819-905c-a0b774ab1b97\") " pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr942\" (UniqueName: \"kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9k4\" (UniqueName: \"kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.554643 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dctq\" (UniqueName: \"kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfdxk\" (UniqueName: \"kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk\") pod \"kas-connection-checker-685895c7c9-4j2vn\" (UID: \"5183f685-7bd0-41b0-a632-9d6e56bc3951\") " pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rgq\" (UniqueName: \"kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2fv\" (UniqueName: \"kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv\") pod \"network-check-source-6f78d59f99-qxx47\" (UID: \"16a3f8c9-121b-49f0-a822-12a395e21af7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:35.554899 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.555281 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.554900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.561208 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.561190 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 04:43:35.562520 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.562496 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 04:43:35.562627 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.562615 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 04:43:35.562892 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.562874 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 04:43:35.592685 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.592659 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:35.592685 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.592661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:35.592858 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.592844 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:35.642165 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.642143 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 04:43:35.642165 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.642157 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\"" Oct 30 04:43:35.642330 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.642142 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\"" Oct 30 04:43:35.642330 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.642140 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 04:43:35.642330 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.642275 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\"" Oct 30 04:43:35.655524 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.655623 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.655623 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.655623 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.655623 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.655759 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.655794 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.655808 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls podName:801eb2b7-bb95-4b2f-95d6-44880fdf062c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.155792377 +0000 UTC m=+31.061760427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls") pod "dns-default-hq5jn" (UID: "801eb2b7-bb95-4b2f-95d6-44880fdf062c") : secret "dns-default-metrics-tls" not found Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l2f\" (UniqueName: \"kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655958 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.655984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8ms\" (UniqueName: \"kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms\") pod \"kas-connection-checker-685895c7c9-9ff65\" (UID: \"93773016-1fde-4819-905c-a0b774ab1b97\") " pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr942\" (UniqueName: \"kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.656106 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9k4\" (UniqueName: \"kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dctq\" (UniqueName: \"kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdxk\" (UniqueName: \"kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk\") pod \"kas-connection-checker-685895c7c9-4j2vn\" (UID: \"5183f685-7bd0-41b0-a632-9d6e56bc3951\") " pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rgq\" (UniqueName: \"kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656331 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656392 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2fv\" (UniqueName: \"kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv\") pod \"network-check-source-6f78d59f99-qxx47\" (UID: \"16a3f8c9-121b-49f0-a822-12a395e21af7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656411 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656414 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656489 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls podName:1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.156471341 +0000 UTC m=+31.062439401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l2bhc" (UID: "1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc") : secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:35.656804 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656492 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656565 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656584 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656624 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656636 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6f8cd844-zgvtd: secret "image-registry-tls" not found Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.656628 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert podName:5dae5dcf-7d46-4eba-a2c8-1193789e3c28 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.156613677 +0000 UTC m=+31.062581738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-84nvd" (UID: "5dae5dcf-7d46-4eba-a2c8-1193789e3c28") : secret "networking-console-plugin-cert" not found Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsx\" (UniqueName: \"kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzpt\" (UniqueName: \"kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt\") pod \"volume-data-source-validator-58c9f5f46d-jxfs5\" (UID: \"e6975d49-682f-42d6-9ff2-21b90112b948\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq48z\" (UniqueName: \"kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z\") pod \"kas-connection-checker-685895c7c9-cddr7\" (UID: \"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92\") " pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.657595 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656960 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.656987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbhw\" (UniqueName: \"kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvhq\" (UniqueName: \"kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj76p\" (UniqueName: \"kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657234 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657242 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.657412 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls podName:ccd92781-ec40-4093-b4f3-5b91519a86b4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.157399231 +0000 UTC m=+31.063367292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls") pod "image-registry-7f6f8cd844-zgvtd" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4") : secret "image-registry-tls" not found Oct 30 04:43:35.658366 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657676 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.657933 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658009 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658025 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfc9b8cc-wb6qg: secret "image-registry-tls" not found Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658064 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls podName:76fbbb93-c186-42f6-ba44-fd5b021e5d82 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.158050697 +0000 UTC m=+31.064018758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls") pod "image-registry-5dfc9b8cc-wb6qg" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82") : secret "image-registry-tls" not found Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658131 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.158120453 +0000 UTC m=+31.064088543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.658284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.658537 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.658604 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658618 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 04:43:35.659048 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.658661 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.158648189 +0000 UTC m=+31.064616239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : secret "router-metrics-certs-default" not found Oct 30 04:43:35.659571 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.659281 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.659838 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.659703 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:35.659991 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:35.659976 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:36.159956671 +0000 UTC m=+31.065924733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:35.660996 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.660968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.661081 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661047 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.661146 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661079 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.661252 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661211 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.661310 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661277 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.661435 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661404 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.661696 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661675 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.662089 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.661912 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:35.662252 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.662229 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.662363 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.662342 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.662458 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.662435 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.662898 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.662874 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.663445 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.663426 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.663601 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.663584 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.664120 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.664101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.664310 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.664292 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.674703 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.674685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.676947 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.676907 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l2f\" (UniqueName: \"kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:35.677370 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.677302 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhsx\" (UniqueName: \"kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:35.679702 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.678643 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzpt\" (UniqueName: \"kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt\") pod \"volume-data-source-validator-58c9f5f46d-jxfs5\" (UID: \"e6975d49-682f-42d6-9ff2-21b90112b948\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:35.679702 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.678980 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq48z\" (UniqueName: \"kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z\") pod \"kas-connection-checker-685895c7c9-cddr7\" (UID: \"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92\") " pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:35.680084 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.679910 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9k4\" (UniqueName: \"kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:35.681514 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.681490 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:35.681642 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.681620 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dctq\" (UniqueName: \"kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.681827 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.681803 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8ms\" (UniqueName: \"kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms\") pod \"kas-connection-checker-685895c7c9-9ff65\" (UID: \"93773016-1fde-4819-905c-a0b774ab1b97\") " pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:35.683152 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.683134 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr942\" (UniqueName: \"kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.684241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.683832 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.684241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.683867 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbhw\" (UniqueName: \"kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:35.684241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.684010 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj76p\" (UniqueName: \"kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.684241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.684009 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvhq\" (UniqueName: \"kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:35.684241 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.684013 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:35.684539 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.684310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2fv\" (UniqueName: \"kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv\") pod \"network-check-source-6f78d59f99-qxx47\" (UID: \"16a3f8c9-121b-49f0-a822-12a395e21af7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:35.684539 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.684359 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rgq\" (UniqueName: \"kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.685078 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.685062 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdxk\" (UniqueName: \"kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk\") pod \"kas-connection-checker-685895c7c9-4j2vn\" (UID: \"5183f685-7bd0-41b0-a632-9d6e56bc3951\") " pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:35.734091 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.734011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:35.754342 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.754322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:35.766176 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.766153 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:35.780116 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.779948 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:35.800410 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.800188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:35.809324 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.806530 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:35.815254 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.815212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:35.836069 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.835890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:35.840602 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.840209 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:35.855493 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:35.853928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:36.042913 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.042892 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j2vn"] Oct 30 04:43:36.044015 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.043993 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5183f685_7bd0_41b0_a632_9d6e56bc3951.slice/crio-3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9 WatchSource:0}: Error finding container 3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9: Status 404 returned error can't find the container with id 3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9 Oct 30 04:43:36.062412 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.062388 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9ff65"] Oct 30 04:43:36.063103 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.063076 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93773016_1fde_4819_905c_a0b774ab1b97.slice/crio-440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2 WatchSource:0}: Error finding container 440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2: Status 404 returned error can't find the container with id 440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2 Oct 30 04:43:36.084612 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.084545 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cddr7"] Oct 30 04:43:36.085656 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.085630 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9c14b8_58c1_4d2b_bd17_a94c6b3bfd92.slice/crio-23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257 WatchSource:0}: Error finding container 23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257: Status 404 returned error can't find the container with id 23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257 Oct 30 04:43:36.093967 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.093942 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94"] Oct 30 04:43:36.100515 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.100481 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11599cb_3adb_4c57_93e5_6a13d5fd50a1.slice/crio-6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1 WatchSource:0}: Error finding container 6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1: Status 404 returned error can't find the container with id 6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1 Oct 30 04:43:36.155711 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.155627 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv"] Oct 30 04:43:36.156944 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.156922 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47"] Oct 30 04:43:36.160591 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.160522 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64040d3f_1de7_42d3_a9aa_6fefa2a926ed.slice/crio-7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58 WatchSource:0}: Error finding container 7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58: Status 404 returned error can't find the container with id 7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58 Oct 30 04:43:36.162337 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.160937 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a3f8c9_121b_49f0_a822_12a395e21af7.slice/crio-de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a WatchSource:0}: Error finding container de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a: Status 404 returned error can't find the container with id de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.165889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166068 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166085 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6f8cd844-zgvtd: secret "image-registry-tls" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166141 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls podName:ccd92781-ec40-4093-b4f3-5b91519a86b4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166120219 +0000 UTC m=+32.072088286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls") pod "image-registry-7f6f8cd844-zgvtd" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4") : secret "image-registry-tls" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166203 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166266 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert podName:5dae5dcf-7d46-4eba-a2c8-1193789e3c28 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166250407 +0000 UTC m=+32.072218453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-84nvd" (UID: "5dae5dcf-7d46-4eba-a2c8-1193789e3c28") : secret "networking-console-plugin-cert" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166314 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166355 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls podName:1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166342153 +0000 UTC m=+32.072310223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l2bhc" (UID: "1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc") : secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166402 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166439 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166427449 +0000 UTC m=+32.072395499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : secret "router-metrics-certs-default" not found Oct 30 04:43:36.166760 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166458 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166469 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfc9b8cc-wb6qg: secret "image-registry-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.166486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166502 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls podName:76fbbb93-c186-42f6-ba44-fd5b021e5d82 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166491091 +0000 UTC m=+32.072459174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls") pod "image-registry-5dfc9b8cc-wb6qg" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82") : secret "image-registry-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166548 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166581 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.16657003 +0000 UTC m=+32.072538082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166641 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166672 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls podName:801eb2b7-bb95-4b2f-95d6-44880fdf062c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166661014 +0000 UTC m=+32.072629067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls") pod "dns-default-hq5jn" (UID: "801eb2b7-bb95-4b2f-95d6-44880fdf062c") : secret "dns-default-metrics-tls" not found Oct 30 04:43:36.167614 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:36.166735 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.166724771 +0000 UTC m=+32.072692851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:36.198301 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.198271 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5"] Oct 30 04:43:36.217626 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.217504 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-d82h8"] Oct 30 04:43:36.219043 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.219015 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d84d597_b1ed_41d3_a4a2_1add913c1125.slice/crio-d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1 WatchSource:0}: Error finding container d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1: Status 404 returned error can't find the container with id d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1 Oct 30 04:43:36.220396 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.220361 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wf2dd"] Oct 30 04:43:36.220933 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.220911 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80515748_a212_4b00_bec6_9b459babaa08.slice/crio-2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93 WatchSource:0}: Error finding container 2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93: Status 404 returned error can't find the container with id 2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93 Oct 30 04:43:36.251207 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.251182 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-79pwp"] Oct 30 04:43:36.252015 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:36.251996 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93c592f_d506_49e4_ba78_bdaa2de3561a.slice/crio-df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681 WatchSource:0}: Error finding container df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681: Status 404 returned error can't find the container with id df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681 Oct 30 04:43:36.859747 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.859704 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cddr7" event={"ID":"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92","Type":"ContainerStarted","Data":"d15556e3a69c63ad4baa31e66998955fefd0f7aa6080c0489fc264de52fde5f8"} Oct 30 04:43:36.860648 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.859767 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cddr7" event={"ID":"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92","Type":"ContainerStarted","Data":"23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257"} Oct 30 04:43:36.864592 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.864565 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9ff65" event={"ID":"93773016-1fde-4819-905c-a0b774ab1b97","Type":"ContainerStarted","Data":"c46be6892453e4c0dff60166fb88eae0c78db7edcfcc0a4d15b4c4cb46225c30"} Oct 30 04:43:36.864699 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.864615 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9ff65" event={"ID":"93773016-1fde-4819-905c-a0b774ab1b97","Type":"ContainerStarted","Data":"440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2"} Oct 30 04:43:36.867750 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.867344 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" event={"ID":"5183f685-7bd0-41b0-a632-9d6e56bc3951","Type":"ContainerStarted","Data":"082a47e818663e1226455fc26ab89e0ee6f703fac0990c0a76fd60d05d9470d1"} Oct 30 04:43:36.867750 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.867399 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" event={"ID":"5183f685-7bd0-41b0-a632-9d6e56bc3951","Type":"ContainerStarted","Data":"3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9"} Oct 30 04:43:36.872571 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.872504 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" event={"ID":"80515748-a212-4b00-bec6-9b459babaa08","Type":"ContainerStarted","Data":"2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93"} Oct 30 04:43:36.874749 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.874029 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" event={"ID":"16a3f8c9-121b-49f0-a822-12a395e21af7","Type":"ContainerStarted","Data":"de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a"} Oct 30 04:43:36.876099 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.876055 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerStarted","Data":"d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1"} Oct 30 04:43:36.880726 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.880691 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-79pwp" event={"ID":"a93c592f-d506-49e4-ba78-bdaa2de3561a","Type":"ContainerStarted","Data":"df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681"} Oct 30 04:43:36.881907 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.881886 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" event={"ID":"e11599cb-3adb-4c57-93e5-6a13d5fd50a1","Type":"ContainerStarted","Data":"6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1"} Oct 30 04:43:36.885265 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.885241 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" event={"ID":"64040d3f-1de7-42d3-a9aa-6fefa2a926ed","Type":"ContainerStarted","Data":"7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58"} Oct 30 04:43:36.887853 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.887826 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" event={"ID":"e6975d49-682f-42d6-9ff2-21b90112b948","Type":"ContainerStarted","Data":"0baadef4e29f5ecd01c5556883c58d9c40419e773263089d17db5c1d76c73cfe"} Oct 30 04:43:36.898177 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.898125 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cddr7" podStartSLOduration=422.898110681 podStartE2EDuration="7m2.898110681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:36.897376371 +0000 UTC m=+31.803344441" watchObservedRunningTime="2026-10-30 04:43:36.898110681 +0000 UTC m=+31.804078751" Oct 30 04:43:36.935300 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:36.935244 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" podStartSLOduration=422.935209798 podStartE2EDuration="7m2.935209798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:36.933511549 +0000 UTC m=+31.839479618" watchObservedRunningTime="2026-10-30 04:43:36.935209798 +0000 UTC m=+31.841177869" Oct 30 04:43:37.173942 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.173901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:37.174149 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.173974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:37.174149 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:37.174149 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:37.174149 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:37.174357 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:37.174357 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:37.174357 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:37.174267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:37.174503 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174403 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:37.174503 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174465 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls podName:1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.174447922 +0000 UTC m=+34.080415974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l2bhc" (UID: "1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc") : secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:37.174873 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174850 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.174833628 +0000 UTC m=+34.080801682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:37.175007 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174912 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 04:43:37.175007 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174936 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.174928682 +0000 UTC m=+34.080896729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : secret "router-metrics-certs-default" not found Oct 30 04:43:37.175007 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174986 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:37.175007 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.174997 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfc9b8cc-wb6qg: secret "image-registry-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175026 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls podName:76fbbb93-c186-42f6-ba44-fd5b021e5d82 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.175016542 +0000 UTC m=+34.080984597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls") pod "image-registry-5dfc9b8cc-wb6qg" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82") : secret "image-registry-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175091 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175121 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.175112025 +0000 UTC m=+34.081080078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175170 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175181 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6f8cd844-zgvtd: secret "image-registry-tls" not found Oct 30 04:43:37.175208 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175210 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls podName:ccd92781-ec40-4093-b4f3-5b91519a86b4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.175199503 +0000 UTC m=+34.081167555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls") pod "image-registry-7f6f8cd844-zgvtd" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4") : secret "image-registry-tls" not found Oct 30 04:43:37.175457 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175274 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 04:43:37.175457 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175301 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls podName:801eb2b7-bb95-4b2f-95d6-44880fdf062c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.175292321 +0000 UTC m=+34.081260373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls") pod "dns-default-hq5jn" (UID: "801eb2b7-bb95-4b2f-95d6-44880fdf062c") : secret "dns-default-metrics-tls" not found Oct 30 04:43:37.175457 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175348 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 04:43:37.175457 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:37.175375 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert podName:5dae5dcf-7d46-4eba-a2c8-1193789e3c28 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:39.175366013 +0000 UTC m=+34.081334061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-84nvd" (UID: "5dae5dcf-7d46-4eba-a2c8-1193789e3c28") : secret "networking-console-plugin-cert" not found Oct 30 04:43:38.900035 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:38.899998 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" event={"ID":"e6975d49-682f-42d6-9ff2-21b90112b948","Type":"ContainerStarted","Data":"cf747dc33044b901309e7bd6eea694391bc7c45636b357d413b7b47fa7b01315"} Oct 30 04:43:38.939443 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:38.937841 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9ff65" podStartSLOduration=424.937820128 podStartE2EDuration="7m4.937820128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:36.971465428 +0000 UTC m=+31.877433498" watchObservedRunningTime="2026-10-30 04:43:38.937820128 +0000 UTC m=+33.843788198" Oct 30 04:43:38.939443 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:38.938145 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" podStartSLOduration=384.852191428 podStartE2EDuration="6m26.938137608s" totalImagesPullingTime="2.08594618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:38.937260221 +0000 UTC m=+33.843228290" watchObservedRunningTime="2026-10-30 04:43:38.938137608 +0000 UTC m=+33.844105673" Oct 30 04:43:39.197161 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:39.197161 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:39.197527 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.197382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:39.202236 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202196 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:39.202357 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202297 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls podName:1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.20227497 +0000 UTC m=+38.108243017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l2bhc" (UID: "1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc") : secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:39.202831 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202808 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 04:43:39.202947 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202882 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 04:43:39.202947 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202933 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.202915972 +0000 UTC m=+38.108884020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : secret "router-metrics-certs-default" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202954 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls podName:801eb2b7-bb95-4b2f-95d6-44880fdf062c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.202943313 +0000 UTC m=+38.108911363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls") pod "dns-default-hq5jn" (UID: "801eb2b7-bb95-4b2f-95d6-44880fdf062c") : secret "dns-default-metrics-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202976 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202988 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6f8cd844-zgvtd: secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.202813 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203027 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls podName:ccd92781-ec40-4093-b4f3-5b91519a86b4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.203016232 +0000 UTC m=+38.108984293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls") pod "image-registry-7f6f8cd844-zgvtd" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4") : secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203035 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfc9b8cc-wb6qg: secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203051 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.203035324 +0000 UTC m=+38.109003377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203090 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls podName:76fbbb93-c186-42f6-ba44-fd5b021e5d82 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.203078375 +0000 UTC m=+38.109046422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls") pod "image-registry-5dfc9b8cc-wb6qg" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82") : secret "image-registry-tls" not found Oct 30 04:43:39.203192 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203118 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 04:43:39.203737 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203245 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert podName:5dae5dcf-7d46-4eba-a2c8-1193789e3c28 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.203232307 +0000 UTC m=+38.109200367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-84nvd" (UID: "5dae5dcf-7d46-4eba-a2c8-1193789e3c28") : secret "networking-console-plugin-cert" not found Oct 30 04:43:39.203737 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203270 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:39.203737 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.203310 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.203300218 +0000 UTC m=+38.109268276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:39.401523 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.400068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:39.401523 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.400241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:39.401523 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.400489 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 04:43:39.401523 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:39.400553 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:44:11.400534277 +0000 UTC m=+66.306502329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : secret "metrics-daemon-secret" not found Oct 30 04:43:39.409203 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.409170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:39.501715 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.501629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:39.504575 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.504552 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:39.513429 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.513399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:39.518862 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.518841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:39.842836 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.842805 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tbzb"] Oct 30 04:43:39.872416 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.872390 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6v95n"] Oct 30 04:43:39.873822 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:39.873791 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce27bdc4_89f2_4eed_a76d_57280fcb0daa.slice/crio-2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed WatchSource:0}: Error finding container 2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed: Status 404 returned error can't find the container with id 2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed Oct 30 04:43:39.906518 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.906478 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6v95n" event={"ID":"ce27bdc4-89f2-4eed-a76d-57280fcb0daa","Type":"ContainerStarted","Data":"2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed"} Oct 30 04:43:39.909283 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:39.909189 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tbzb" event={"ID":"bcf81c88-4fb2-4655-bd29-f2114fad2d3a","Type":"ContainerStarted","Data":"984b208dbfd4271b46c60da19000bc655682bf0bfe757f166ae2081385e72877"} Oct 30 04:43:40.914429 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:40.914382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/0.log" Oct 30 04:43:40.914889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:40.914444 2562 generic.go:356] "Generic (PLEG): container finished" podID="80515748-a212-4b00-bec6-9b459babaa08" containerID="8d1e57fbc3995f1cf966f036b2d9bf63dfba206f8a8d91dd72baea23b699a3d3" exitCode=255 Oct 30 04:43:40.914889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:40.914489 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" event={"ID":"80515748-a212-4b00-bec6-9b459babaa08","Type":"ContainerDied","Data":"8d1e57fbc3995f1cf966f036b2d9bf63dfba206f8a8d91dd72baea23b699a3d3"} Oct 30 04:43:40.914889 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:40.914852 2562 scope.go:122] "RemoveContainer" containerID="8d1e57fbc3995f1cf966f036b2d9bf63dfba206f8a8d91dd72baea23b699a3d3" Oct 30 04:43:41.924960 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.924375 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" event={"ID":"e11599cb-3adb-4c57-93e5-6a13d5fd50a1","Type":"ContainerStarted","Data":"db5d6ce4071701bca64fc44717d93486526933dc4be82a7d0cb80b685ba5c3a4"} Oct 30 04:43:41.929240 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.928827 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" event={"ID":"64040d3f-1de7-42d3-a9aa-6fefa2a926ed","Type":"ContainerStarted","Data":"de88270bb0d30814b8c5cf9cd7d331ce7a22408d8d64401f1f4b3d3da592e0e2"} Oct 30 04:43:41.931568 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.931498 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.932078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/0.log" Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.932111 2562 generic.go:356] "Generic (PLEG): container finished" podID="80515748-a212-4b00-bec6-9b459babaa08" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" exitCode=255 Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.932174 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" event={"ID":"80515748-a212-4b00-bec6-9b459babaa08","Type":"ContainerDied","Data":"8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f"} Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.932213 2562 scope.go:122] "RemoveContainer" containerID="8d1e57fbc3995f1cf966f036b2d9bf63dfba206f8a8d91dd72baea23b699a3d3" Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.932565 2562 scope.go:122] "RemoveContainer" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" Oct 30 04:43:41.932949 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:41.932729 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:41.936749 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.936357 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" event={"ID":"16a3f8c9-121b-49f0-a822-12a395e21af7","Type":"ContainerStarted","Data":"579115e462cbc20cf3ed999005405d477b68d136a69a7ab6699633dbfe4dc2e2"} Oct 30 04:43:41.940495 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:41.940441 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerStarted","Data":"591e5fe34fea82975462c61acf9be24977336dfe766991ed4abbe97f80ea7084"} Oct 30 04:43:42.003557 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.003485 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" podStartSLOduration=387.85010643 podStartE2EDuration="6m33.003464752s" totalImagesPullingTime="5.153358322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:41.966424691 +0000 UTC m=+36.872392762" watchObservedRunningTime="2026-10-30 04:43:42.003464752 +0000 UTC m=+36.909432821" Oct 30 04:43:42.043043 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.042471 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" podStartSLOduration=388.531108326 podStartE2EDuration="6m34.042456534s" totalImagesPullingTime="5.511348208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:42.041680379 +0000 UTC m=+36.947648450" watchObservedRunningTime="2026-10-30 04:43:42.042456534 +0000 UTC m=+36.948424604" Oct 30 04:43:42.051537 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.051484 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" podStartSLOduration=391.570711301 podStartE2EDuration="6m37.05146655s" totalImagesPullingTime="5.480755249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:42.004194511 +0000 UTC m=+36.910162585" watchObservedRunningTime="2026-10-30 04:43:42.05146655 +0000 UTC m=+36.957434621" Oct 30 04:43:42.143369 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.142433 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-d82h8" podStartSLOduration=385.092269478 podStartE2EDuration="6m30.142414741s" totalImagesPullingTime="5.050145263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:42.0934901 +0000 UTC m=+36.999458170" watchObservedRunningTime="2026-10-30 04:43:42.142414741 +0000 UTC m=+37.048382814" Oct 30 04:43:42.674158 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.674118 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2bq"] Oct 30 04:43:42.698463 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.698411 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2bq"] Oct 30 04:43:42.698670 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.698583 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.735671 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.735626 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 04:43:42.753926 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.753893 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:43:42.769065 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.769032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:42.780171 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.777768 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:43:42.790574 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.785741 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 04:43:42.790574 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.785977 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\"" Oct 30 04:43:42.790574 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.786130 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wfdkq\"" Oct 30 04:43:42.837715 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.837626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.837881 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.837811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.837881 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.837853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.837982 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.837958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.838059 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.837994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspx8\" (UniqueName: \"kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939192 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:42.939420 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:42.939478 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.439458997 +0000 UTC m=+38.345427063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : configmap "kube-rbac-proxy" not found Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939488 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939565 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.939656 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:42.939662 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:42.940187 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:42.940187 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:42.939708 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:43.439693547 +0000 UTC m=+38.345661609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:42.940187 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.939746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspx8\" (UniqueName: \"kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.953378 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.953342 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-79pwp" event={"ID":"a93c592f-d506-49e4-ba78-bdaa2de3561a","Type":"ContainerStarted","Data":"461644985c100779aa87ab3f245264df5f2ab0a40eb2a820f651a892ae324957"} Oct 30 04:43:42.958452 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.958422 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:42.959046 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.959025 2562 scope.go:122] "RemoveContainer" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" Oct 30 04:43:42.959237 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:42.959194 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:42.961076 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.961046 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspx8\" (UniqueName: \"kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:42.993026 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:42.992971 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-79pwp" podStartSLOduration=2.209055965 podStartE2EDuration="7.99295383s" totalImagesPullingTime="5.783897865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:42.99143349 +0000 UTC m=+37.897401586" watchObservedRunningTime="2026-10-30 04:43:42.99295383 +0000 UTC m=+37.898921937" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.041026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.041073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.041189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.041240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.041866 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.042796 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.043243 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.043187 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.064021 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.063994 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.101019 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.100943 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:43.236165 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.236088 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v"] Oct 30 04:43:43.250039 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250162 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250177 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6f8cd844-zgvtd: secret "image-registry-tls" not found Oct 30 04:43:43.250188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250247 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls podName:ccd92781-ec40-4093-b4f3-5b91519a86b4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250229979 +0000 UTC m=+46.156198042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls") pod "image-registry-7f6f8cd844-zgvtd" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4") : secret "image-registry-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250261 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250325 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls podName:801eb2b7-bb95-4b2f-95d6-44880fdf062c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250306701 +0000 UTC m=+46.156274767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls") pod "dns-default-hq5jn" (UID: "801eb2b7-bb95-4b2f-95d6-44880fdf062c") : secret "dns-default-metrics-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250328 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.250350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250366 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250378 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfc9b8cc-wb6qg: secret "image-registry-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250382 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls podName:1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250366738 +0000 UTC m=+46.156334802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l2bhc" (UID: "1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc") : secret "cluster-monitoring-operator-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250424 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls podName:76fbbb93-c186-42f6-ba44-fd5b021e5d82 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250411164 +0000 UTC m=+46.156379227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls") pod "image-registry-5dfc9b8cc-wb6qg" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82") : secret "image-registry-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250424 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250457 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250448901 +0000 UTC m=+46.156416993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:43.250487 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250475 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 04:43:43.251130 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250501 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250490858 +0000 UTC m=+46.156458957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:43.251130 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250512 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 04:43:43.251130 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250520 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert podName:5dae5dcf-7d46-4eba-a2c8-1193789e3c28 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250510713 +0000 UTC m=+46.156478762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-84nvd" (UID: "5dae5dcf-7d46-4eba-a2c8-1193789e3c28") : secret "networking-console-plugin-cert" not found Oct 30 04:43:43.251130 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.250550 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.250541051 +0000 UTC m=+46.156509125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : secret "router-metrics-certs-default" not found Oct 30 04:43:43.270206 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.270178 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9"] Oct 30 04:43:43.270710 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.270684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:43.284141 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.284119 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v"] Oct 30 04:43:43.284268 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.284166 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9"] Oct 30 04:43:43.284326 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.284274 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:43.288542 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.287055 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:43.288542 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.287072 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tqz9q\"" Oct 30 04:43:43.288542 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.287206 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 04:43:43.410777 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.408600 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nxbx2"] Oct 30 04:43:43.426555 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.426478 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.426715 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.426604 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nxbx2"] Oct 30 04:43:43.429655 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.429629 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:43:43.438820 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:43.438794 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c4081d_a7a9_4c3b_8f6d_bd50698a6b88.slice/crio-9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b WatchSource:0}: Error finding container 9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b: Status 404 returned error can't find the container with id 9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b Oct 30 04:43:43.441622 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.441599 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 04:43:43.441711 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.441683 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 04:43:43.441766 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.441719 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 04:43:43.441766 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.441728 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 04:43:43.441892 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.441784 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dz97m\"" Oct 30 04:43:43.452235 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.452193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:43.452326 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.452300 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 04:43:43.452326 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.452316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:43.452493 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.452368 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:44.452349504 +0000 UTC m=+39.358317567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : configmap "kube-rbac-proxy" not found Oct 30 04:43:43.452493 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.452432 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:43.452493 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.452470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j477b\" (UniqueName: \"kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b\") pod \"migrator-66df5d496b-zt75v\" (UID: \"115a2fb1-3568-46e3-90a2-98a88c00c27c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:43.452493 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:43.452494 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:44.452476885 +0000 UTC m=+39.358444947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:43.452674 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.452558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgqr\" (UniqueName: \"kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr\") pod \"migrator-66df5d496b-kgcn9\" (UID: \"ef12024a-04a5-41ff-8bfc-18ddd645d296\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:43.553825 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.553794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.553952 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.553928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.554188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.554166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j477b\" (UniqueName: \"kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b\") pod \"migrator-66df5d496b-zt75v\" (UID: \"115a2fb1-3568-46e3-90a2-98a88c00c27c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:43.554327 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.554231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgqr\" (UniqueName: \"kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr\") pod \"migrator-66df5d496b-kgcn9\" (UID: \"ef12024a-04a5-41ff-8bfc-18ddd645d296\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:43.554327 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.554300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ff4\" (UniqueName: \"kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.573207 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.573179 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j477b\" (UniqueName: \"kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b\") pod \"migrator-66df5d496b-zt75v\" (UID: \"115a2fb1-3568-46e3-90a2-98a88c00c27c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:43.573406 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.573388 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgqr\" (UniqueName: \"kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr\") pod \"migrator-66df5d496b-kgcn9\" (UID: \"ef12024a-04a5-41ff-8bfc-18ddd645d296\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:43.599011 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.598986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:43.621496 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.621469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:43.655409 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.655124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ff4\" (UniqueName: \"kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.655409 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.655207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.655409 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.655291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.656096 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.656059 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.658305 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.658284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.677582 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.677561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ff4\" (UniqueName: \"kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.765641 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.765607 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:43.940192 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:43.940158 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115a2fb1_3568_46e3_90a2_98a88c00c27c.slice/crio-28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666 WatchSource:0}: Error finding container 28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666: Status 404 returned error can't find the container with id 28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666 Oct 30 04:43:43.940779 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.940351 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v"] Oct 30 04:43:43.962974 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.962938 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" event={"ID":"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88","Type":"ContainerStarted","Data":"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6"} Oct 30 04:43:43.963096 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.962993 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" event={"ID":"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88","Type":"ContainerStarted","Data":"9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b"} Oct 30 04:43:43.965082 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.965056 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6v95n" event={"ID":"ce27bdc4-89f2-4eed-a76d-57280fcb0daa","Type":"ContainerStarted","Data":"e76063e3c3f14a08001d3f01840347dbbd0343733c00fcff0e918041d449eda6"} Oct 30 04:43:43.965237 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.965199 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:43.967134 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.967111 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" event={"ID":"115a2fb1-3568-46e3-90a2-98a88c00c27c","Type":"ContainerStarted","Data":"28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666"} Oct 30 04:43:43.970803 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:43.970781 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9"] Oct 30 04:43:43.971934 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:43.971912 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef12024a_04a5_41ff_8bfc_18ddd645d296.slice/crio-ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871 WatchSource:0}: Error finding container ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871: Status 404 returned error can't find the container with id ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871 Oct 30 04:43:44.005065 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.005013 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" podStartSLOduration=2.004996954 podStartE2EDuration="2.004996954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:44.002846552 +0000 UTC m=+38.908814639" watchObservedRunningTime="2026-10-30 04:43:44.004996954 +0000 UTC m=+38.910965023" Oct 30 04:43:44.041251 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.041129 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6v95n" podStartSLOduration=34.262892863 podStartE2EDuration="38.041116132s" totalImagesPullingTime="3.778223269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:44.039851909 +0000 UTC m=+38.945819978" watchObservedRunningTime="2026-10-30 04:43:44.041116132 +0000 UTC m=+38.947084202" Oct 30 04:43:44.078991 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.078926 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nxbx2"] Oct 30 04:43:44.079101 ip-10-0-139-228 kubenswrapper[2562]: W1030 04:43:44.079044 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2e0ccd_b10f_43ed_a75d_3672190d3d6e.slice/crio-7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7 WatchSource:0}: Error finding container 7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7: Status 404 returned error can't find the container with id 7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7 Oct 30 04:43:44.461958 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.461895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:44.462273 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:44.462037 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 04:43:44.462273 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:44.462115 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:46.462093032 +0000 UTC m=+41.368061096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : configmap "kube-rbac-proxy" not found Oct 30 04:43:44.462273 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.462193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:44.462477 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:44.462452 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:44.462550 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:44.462508 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:46.462491437 +0000 UTC m=+41.368459506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:44.972569 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.972539 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" event={"ID":"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e","Type":"ContainerStarted","Data":"e107c614c42fe181d63aa052f5dc0e6633f95a53bea8b01e43b050f196142b0b"} Oct 30 04:43:44.973058 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.972593 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" event={"ID":"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e","Type":"ContainerStarted","Data":"7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7"} Oct 30 04:43:44.975188 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:44.975124 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" event={"ID":"ef12024a-04a5-41ff-8bfc-18ddd645d296","Type":"ContainerStarted","Data":"ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871"} Oct 30 04:43:45.840617 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.840589 2562 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:45.841387 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.840845 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:45.841711 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.841693 2562 scope.go:122] "RemoveContainer" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" Oct 30 04:43:45.841965 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:45.841934 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:45.934337 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.934315 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9k5j_37948e1b-b416-4df5-9c56-7ff2e7996e48/dns-node-resolver/0.log" Oct 30 04:43:45.983778 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.983742 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" event={"ID":"115a2fb1-3568-46e3-90a2-98a88c00c27c","Type":"ContainerStarted","Data":"ce38caf0dfd9a53ba43faae8df6fc7ba792967876997ba457bea83ed3e31d568"} Oct 30 04:43:45.987485 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:45.987415 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" event={"ID":"ef12024a-04a5-41ff-8bfc-18ddd645d296","Type":"ContainerStarted","Data":"311435147ef14320f77c2da07bba80262ffba899fd29a061b54c42fe04d32eba"} Oct 30 04:43:46.022277 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.021508 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" podStartSLOduration=3.021489156 podStartE2EDuration="3.021489156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:45.013171069 +0000 UTC m=+39.919139145" watchObservedRunningTime="2026-10-30 04:43:46.021489156 +0000 UTC m=+40.927457222" Oct 30 04:43:46.023030 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.022991 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" podStartSLOduration=1.167146261 podStartE2EDuration="3.022981278s" totalImagesPullingTime="1.855835017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:46.02096618 +0000 UTC m=+40.926934242" watchObservedRunningTime="2026-10-30 04:43:46.022981278 +0000 UTC m=+40.928949361" Oct 30 04:43:46.059503 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.059399 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" podStartSLOduration=1.234005622 podStartE2EDuration="3.059379313s" totalImagesPullingTime="1.825373691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:46.057236421 +0000 UTC m=+40.963204484" watchObservedRunningTime="2026-10-30 04:43:46.059379313 +0000 UTC m=+40.965347385" Oct 30 04:43:46.486548 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.486514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:46.486721 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.486615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:46.486787 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:46.486765 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:46.486847 ip-10-0-139-228 kubenswrapper[2562]: E1030 04:43:46.486836 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:50.486821737 +0000 UTC m=+45.392789800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:46.487114 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.487096 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:46.933651 ip-10-0-139-228 kubenswrapper[2562]: I1030 04:43:46.933622 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lphjh_bbaf9947-5532-47ce-aea5-dcb456c37684/node-ca/0.log" Oct 30 04:43:46.968600 ip-10-0-139-228 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 04:43:46.986611 ip-10-0-139-228 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 04:43:46.986783 ip-10-0-139-228 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 04:43:46.986975 ip-10-0-139-228 systemd[1]: kubelet.service: Consumed 3.224s CPU time. Oct 30 04:43:46.999854 ip-10-0-139-228 systemd[1]: Starting Kubernetes Kubelet... Oct 30 04:43:47.074550 ip-10-0-139-228 kubenswrapper[5058]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:47.074550 ip-10-0-139-228 kubenswrapper[5058]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 04:43:47.074550 ip-10-0-139-228 kubenswrapper[5058]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:47.074550 ip-10-0-139-228 kubenswrapper[5058]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:47.076579 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076493 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.076579 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076579 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076585 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076588 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076593 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076597 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076600 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076603 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076606 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076609 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076612 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076614 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076617 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076628 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076631 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076634 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076637 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076639 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076642 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076645 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.076648 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076648 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076651 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076654 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076656 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076659 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076662 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076665 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076668 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076670 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076673 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076675 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076678 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076681 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076684 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076686 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076689 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076691 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076694 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076697 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.077125 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076699 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076701 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076704 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076707 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076710 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076713 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076716 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076719 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076722 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076725 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076728 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076730 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076733 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076735 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076738 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076741 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076744 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076746 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076749 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076752 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.077631 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076755 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076757 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076760 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076762 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076765 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076767 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076770 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076773 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076776 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076779 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076781 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076784 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076787 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076789 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076792 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076794 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076797 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076799 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076802 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076805 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.078131 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076813 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076816 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076819 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076821 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076824 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076826 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076830 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076833 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076835 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076838 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076840 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076843 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076846 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076851 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076854 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076857 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076860 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076863 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076870 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.078659 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076873 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076875 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076878 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076881 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076883 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076886 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076888 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.076891 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077698 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077706 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077710 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077713 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077716 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077719 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077728 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077731 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077734 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077736 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077739 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077742 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.079135 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077745 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077747 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077750 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077753 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077755 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077759 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077761 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077764 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077767 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077769 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077772 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077775 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077777 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077780 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077783 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077786 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077788 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077791 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077793 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077796 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.079670 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077799 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077801 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077804 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077806 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077809 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077811 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077813 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077822 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077824 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077827 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077829 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077832 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077835 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077838 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077841 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077844 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077848 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077851 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077855 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.080158 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077859 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077863 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077867 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077870 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077874 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077878 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077882 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077885 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077888 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077891 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077893 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077896 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077909 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077914 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077917 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077920 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077923 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077925 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077928 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.080703 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077930 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077933 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077949 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077952 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077954 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077957 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077960 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077963 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077966 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077968 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077971 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077974 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077977 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077979 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077982 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077985 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077987 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077990 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077993 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.077997 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.081176 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078000 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078003 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078007 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078010 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078012 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078015 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078017 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078020 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078022 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078025 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078027 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078030 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078033 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078035 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078038 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078041 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078551 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078558 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078561 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.081692 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078564 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078567 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078569 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078572 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078575 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078578 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078580 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078583 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078585 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078588 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078591 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078593 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078596 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078599 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078601 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078604 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078607 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078609 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078612 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078614 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.082229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078617 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078622 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078625 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078628 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078630 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078633 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078636 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078638 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078641 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078644 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078647 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078650 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078653 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078655 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078658 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078660 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078663 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078667 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078670 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.082754 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078673 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078676 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078679 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078681 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078684 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078687 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078689 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078692 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078695 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078697 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078700 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078702 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078705 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078707 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078712 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078715 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078717 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078720 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078722 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.083448 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078725 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078727 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078730 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078733 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078736 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078739 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078741 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078746 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078750 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078754 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078757 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078760 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078763 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078766 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078769 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078772 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078774 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078777 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078779 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.084148 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078782 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078784 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078787 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078790 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078792 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078795 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078797 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078800 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078804 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078806 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078809 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078812 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078814 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078817 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078820 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078822 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078825 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078828 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078831 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078833 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.084756 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078836 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078838 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078842 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078844 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078847 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.078849 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078940 5058 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078947 5058 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078959 5058 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078964 5058 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078969 5058 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078972 5058 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078979 5058 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078983 5058 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078986 5058 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078989 5058 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078993 5058 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.078996 5058 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079000 5058 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079003 5058 flags.go:64] FLAG: --cgroup-root="" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079007 5058 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079010 5058 flags.go:64] FLAG: --client-ca-file="" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079013 5058 flags.go:64] FLAG: --cloud-provider="external" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079016 5058 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 04:43:47.085282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079021 5058 flags.go:64] FLAG: --cluster-domain="" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079024 5058 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079027 5058 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079030 5058 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079035 5058 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079039 5058 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079043 5058 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079046 5058 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079050 5058 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079053 5058 flags.go:64] FLAG: --contention-profiling="false" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079056 5058 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079059 5058 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079062 5058 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079065 5058 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079069 5058 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079073 5058 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079075 5058 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079078 5058 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079081 5058 flags.go:64] FLAG: --enable-server="true" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079084 5058 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079089 5058 flags.go:64] FLAG: --event-burst="100" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079092 5058 flags.go:64] FLAG: --event-qps="50" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079095 5058 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079098 5058 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 04:43:47.085890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079101 5058 flags.go:64] FLAG: --eviction-hard="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079106 5058 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079109 5058 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079112 5058 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079117 5058 flags.go:64] FLAG: --eviction-soft="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079120 5058 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079123 5058 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079126 5058 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079129 5058 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079132 5058 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079136 5058 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079139 5058 flags.go:64] FLAG: --feature-gates="" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079148 5058 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079152 5058 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079155 5058 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079158 5058 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079162 5058 flags.go:64] FLAG: --healthz-port="10248" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079165 5058 flags.go:64] FLAG: --help="false" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079168 5058 flags.go:64] FLAG: --hostname-override="ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079171 5058 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079174 5058 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079177 5058 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079180 5058 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 04:43:47.086509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079184 5058 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079187 5058 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079190 5058 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079193 5058 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079196 5058 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079199 5058 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079203 5058 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079205 5058 flags.go:64] FLAG: --kube-reserved="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079208 5058 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079212 5058 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079232 5058 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079235 5058 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079238 5058 flags.go:64] FLAG: --lock-file="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079243 5058 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079246 5058 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079249 5058 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079255 5058 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079258 5058 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079261 5058 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079264 5058 flags.go:64] FLAG: --logging-format="text" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079267 5058 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079272 5058 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079275 5058 flags.go:64] FLAG: --manifest-url="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079278 5058 flags.go:64] FLAG: --manifest-url-header="" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079283 5058 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 04:43:47.087074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079287 5058 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079291 5058 flags.go:64] FLAG: --max-pods="110" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079294 5058 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079297 5058 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079301 5058 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079304 5058 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079307 5058 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079310 5058 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079313 5058 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079320 5058 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079324 5058 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079327 5058 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079330 5058 flags.go:64] FLAG: --pod-cidr="" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079333 5058 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079336 5058 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079339 5058 flags.go:64] FLAG: --pods-per-core="0" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079342 5058 flags.go:64] FLAG: --port="10250" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079345 5058 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079348 5058 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0789f657a9600d716" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079352 5058 flags.go:64] FLAG: --qos-reserved="" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079356 5058 flags.go:64] FLAG: --read-only-port="10255" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079360 5058 flags.go:64] FLAG: --register-node="true" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079362 5058 flags.go:64] FLAG: --register-with-taints="" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079366 5058 flags.go:64] FLAG: --registry-burst="10" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079369 5058 flags.go:64] FLAG: --registry-qps="5" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079372 5058 flags.go:64] FLAG: --reserved-cpus="" Oct 30 04:43:47.087929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079375 5058 flags.go:64] FLAG: --reserved-memory="" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079379 5058 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079383 5058 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079386 5058 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079389 5058 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079393 5058 flags.go:64] FLAG: --runonce="false" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079396 5058 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079399 5058 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079403 5058 flags.go:64] FLAG: --seccomp-default="false" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079406 5058 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079409 5058 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079412 5058 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079415 5058 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079418 5058 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079421 5058 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079424 5058 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079427 5058 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079430 5058 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079433 5058 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079437 5058 flags.go:64] FLAG: --system-cgroups="" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079439 5058 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079445 5058 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079448 5058 flags.go:64] FLAG: --tls-cert-file="" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079450 5058 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079455 5058 flags.go:64] FLAG: --tls-min-version="" Oct 30 04:43:47.088607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079458 5058 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079462 5058 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079465 5058 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079468 5058 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079471 5058 flags.go:64] FLAG: --v="2" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079476 5058 flags.go:64] FLAG: --version="false" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079480 5058 flags.go:64] FLAG: --vmodule="" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079484 5058 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079487 5058 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079619 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079622 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079626 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079629 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079632 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079635 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079638 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079641 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079644 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079646 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079649 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079652 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.089332 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079655 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079657 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079661 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079665 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079668 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079671 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079674 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079677 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079680 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079683 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079686 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079688 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079692 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079695 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079697 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079700 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079704 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079708 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079711 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.089848 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079714 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079718 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079720 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079723 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079726 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079728 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079731 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079734 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079737 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079739 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079742 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079744 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079747 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079750 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079752 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079755 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079757 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079760 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079763 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.090334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079765 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079768 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079770 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079773 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079775 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079778 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079782 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079785 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079788 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079790 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079793 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079796 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079798 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079801 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079805 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079808 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079810 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079813 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079815 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079818 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.090815 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079821 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079823 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079826 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079830 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079832 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079835 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079838 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079840 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079843 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079846 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079848 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079851 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079853 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079856 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079859 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079861 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079864 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079866 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079870 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079873 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.091328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079876 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079878 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079881 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079883 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079886 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079889 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079892 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079895 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079898 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079900 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079903 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079905 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079908 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079911 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079913 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079916 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.091814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.079941 5058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079977 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079982 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079985 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079988 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079991 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079994 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079997 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.079999 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080002 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080005 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080007 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080010 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080013 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080015 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080020 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080024 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080028 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080031 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.092229 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080034 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080037 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080040 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080044 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080046 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080049 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080052 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080054 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080057 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080059 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080063 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080067 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080069 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080072 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080075 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080078 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080080 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080083 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080086 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.092706 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080088 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080091 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080094 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080096 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080099 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080101 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080104 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080107 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080109 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080113 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080116 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080118 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080121 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080124 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080126 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080129 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080134 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080136 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080139 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080142 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.093168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080145 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080148 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080150 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080153 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080156 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080159 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080162 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080165 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080168 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080171 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080173 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080176 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080179 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080181 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080184 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080186 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080189 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080192 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080194 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080197 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.093825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080200 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080203 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080206 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080208 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080211 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080228 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080232 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080235 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080239 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080242 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080245 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080247 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080250 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080253 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080255 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080258 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080261 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080264 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080266 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080269 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.094334 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080272 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080275 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080278 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080281 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080284 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080286 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080289 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080291 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.080294 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.085746 5058 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.085764 5058 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.085787 5058 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.085793 5058 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085854 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085859 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085862 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085866 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085869 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085872 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.094821 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085875 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085877 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085880 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085884 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085886 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085889 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085892 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085895 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085898 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085901 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085903 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085906 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085909 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085911 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085914 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085917 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085920 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085923 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085926 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085929 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.095335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085932 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085934 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085940 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085943 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085946 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085949 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085952 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085954 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085957 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085960 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085962 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085965 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085967 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085970 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085973 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085975 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085978 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085981 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085984 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.095814 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085987 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085995 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.085998 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086001 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086003 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086006 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086009 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086012 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086015 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086018 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086021 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086024 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086026 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086029 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086032 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086035 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086038 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086040 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086043 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086045 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.096297 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086048 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086050 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086053 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086055 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086075 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086079 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086083 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086086 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086088 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086091 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086093 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086096 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086099 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086101 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086109 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086112 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086116 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086120 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086122 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.096782 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086125 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086128 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086131 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086134 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086137 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086139 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086141 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086144 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086147 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086149 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086152 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086154 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086157 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086159 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086163 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086167 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086170 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086173 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086177 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.097526 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086180 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086183 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086186 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.086214 5058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086364 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086368 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086371 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086374 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086376 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086387 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086390 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086392 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086395 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086397 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086400 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086404 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.098067 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086406 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086410 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086414 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086417 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086420 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086423 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086426 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086428 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086431 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086433 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086436 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086439 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086441 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086444 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086446 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086449 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086451 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086455 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086459 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.098540 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086463 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086465 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086468 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086471 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086473 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086476 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086479 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086482 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086485 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086488 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086490 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086493 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086496 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086498 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086501 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086504 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086507 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086509 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086512 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086515 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.099074 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086518 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086521 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086524 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086526 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086529 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086532 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086535 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086538 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086543 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086547 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086551 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086555 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086572 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086574 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086577 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086580 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086582 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086585 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086588 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.099662 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086590 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086595 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086597 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086600 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086603 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086606 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086608 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086611 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086614 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086616 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086619 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086621 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086624 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086626 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086629 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086632 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086635 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086637 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086640 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086643 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.100296 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086645 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086648 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086650 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086653 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086655 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086660 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086662 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086665 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086667 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086670 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086673 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086676 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086678 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086681 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086685 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086688 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.100939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.086714 5058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086747 5058 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086751 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086754 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086757 5058 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086760 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086762 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086765 5058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086768 5058 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086770 5058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086773 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086776 5058 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086778 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086781 5058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086784 5058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086787 5058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086790 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086792 5058 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086795 5058 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086797 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:47.101405 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086800 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086803 5058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086807 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086809 5058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086812 5058 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086815 5058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086818 5058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086820 5058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086823 5058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086826 5058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086828 5058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086831 5058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086839 5058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086843 5058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086846 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086850 5058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086853 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086855 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086858 5058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:47.101945 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086861 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086863 5058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086866 5058 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086868 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086871 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086874 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086876 5058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086879 5058 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086881 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086884 5058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086888 5058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086891 5058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086895 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086898 5058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086901 5058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086905 5058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086908 5058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086911 5058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086913 5058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086916 5058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:47.102428 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086919 5058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086922 5058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086924 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086927 5058 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086929 5058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086934 5058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086937 5058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086940 5058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086943 5058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086945 5058 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086948 5058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086951 5058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086953 5058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086956 5058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086959 5058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086961 5058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086964 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086966 5058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086969 5058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086971 5058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:47.102914 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086974 5058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086976 5058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086979 5058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086981 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086984 5058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086986 5058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086989 5058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086993 5058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086996 5058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.086998 5058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087001 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087003 5058 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087006 5058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087009 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087011 5058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087014 5058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087016 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087020 5058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087029 5058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:47.103475 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087032 5058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087034 5058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087036 5058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087039 5058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087042 5058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087044 5058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087047 5058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087050 5058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:47.087053 5058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.087108 5058 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088126 5058 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088175 5058 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088362 5058 server.go:1014] "Starting client certificate rotation" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088503 5058 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088555 5058 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:06 +0000 UTC" deadline="2028-07-19 13:37:56.668110948 +0000 UTC" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088591 5058 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15080h54m9.57952222s" Oct 30 04:43:47.103935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.088829 5058 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:47.104494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.097049 5058 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:47.104494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.098257 5058 log.go:25] "Validated CRI v1 runtime API" Oct 30 04:43:47.104494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.099985 5058 log.go:25] "Validated CRI v1 image API" Oct 30 04:43:47.104494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.100675 5058 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 04:43:47.104494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.102646 5058 fs.go:135] Filesystem UUIDs: map[2656cefc-8d8a-434f-aa8a-8d786b3cd2b8:/dev/nvme0n1p4 3542657d-cc51-4bd7-ac7b-19cb06f630dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 04:43:47.104653 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.102669 5058 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0baadef4e29f5ecd01c5556883c58d9c40419e773263089d17db5c1d76c73cfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0baadef4e29f5ecd01c5556883c58d9c40419e773263089d17db5c1d76c73cfe/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/984b208dbfd4271b46c60da19000bc655682bf0bfe757f166ae2081385e72877/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/984b208dbfd4271b46c60da19000bc655682bf0bfe757f166ae2081385e72877/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e584bc-955f-4e38-a161-0258bdb03d24/volumes/kubernetes.io~projected/kube-api-access-qspx8:{mountpoint:/var/lib/kubelet/pods/09e584bc-955f-4e38-a161-0258bdb03d24/volumes/kubernetes.io~projected/kube-api-access-qspx8 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/115a2fb1-3568-46e3-90a2-98a88c00c27c/volumes/kubernetes.io~projected/kube-api-access-j477b:{mountpoint:/var/lib/kubelet/pods/115a2fb1-3568-46e3-90a2-98a88c00c27c/volumes/kubernetes.io~projected/kube-api-access-j477b major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16a3f8c9-121b-49f0-a822-12a395e21af7/volumes/kubernetes.io~projected/kube-api-access-zj2fv:{mountpoint:/var/lib/kubelet/pods/16a3f8c9-121b-49f0-a822-12a395e21af7/volumes/kubernetes.io~projected/kube-api-access-zj2fv major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc/volumes/kubernetes.io~projected/kube-api-access-x5l2f:{mountpoint:/var/lib/kubelet/pods/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc/volumes/kubernetes.io~projected/kube-api-access-x5l2f major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88/volumes/kubernetes.io~projected/kube-api-access-b99km:{mountpoint:/var/lib/kubelet/pods/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88/volumes/kubernetes.io~projected/kube-api-access-b99km major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37948e1b-b416-4df5-9c56-7ff2e7996e48/volumes/kubernetes.io~projected/kube-api-access-5tjz5:{mountpoint:/var/lib/kubelet/pods/37948e1b-b416-4df5-9c56-7ff2e7996e48/volumes/kubernetes.io~projected/kube-api-access-5tjz5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d95e075-6720-45ab-ab71-f8b5bc189697/volumes/kubernetes.io~projected/kube-api-access-5l9z2:{mountpoint:/var/lib/kubelet/pods/3d95e075-6720-45ab-ab71-f8b5bc189697/volumes/kubernetes.io~projected/kube-api-access-5l9z2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~projected/kube-api-access-bzhsx:{mountpoint:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~projected/kube-api-access-bzhsx major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5183f685-7bd0-41b0-a632-9d6e56bc3951/volumes/kubernetes.io~projected/kube-api-access-lfdxk:{mountpoint:/var/lib/kubelet/pods/5183f685-7bd0-41b0-a632-9d6e56bc3951/volumes/kubernetes.io~projected/kube-api-access-lfdxk major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5771f103-055c-43b2-b344-fb3f64cc0998/volumes/kubernetes.io~projected/kube-api-access-464tl:{mountpoint:/var/lib/kubelet/pods/5771f103-055c-43b2-b344-fb3f64cc0998/volumes/kubernetes.io~projected/kube-api-access-464tl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~projected/kube-api-access-6dctq:{mountpoint:/var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~projected/kube-api-access-6dctq major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~projected/kube-api-access-99ff4:{mountpoint:/var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~projected/kube-api-access-99ff4 major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92/volumes/kubernetes.io~projected/kube-api-access-hq48z:{mountpoint:/var/lib/kubelet/pods/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92/volumes/kubernetes.io~projected/kube-api-access-hq48z major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/kube-api-access-79bn8:{mountpoint:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/kube-api-access-79bn8 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801eb2b7-bb95-4b2f-95d6-44880fdf062c/volumes/kubernetes.io~projected/kube-api-access-npbhw:{mountpoint:/var/lib/kubelet/pods/801eb2b7-bb95-4b2f-95d6-44880fdf062c/volumes/kubernetes.io~projected/kube-api-access-npbhw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~projected/kube-api-access-rj76p:{mountpoint:/var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~projected/kube-api-access-rj76p major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a79724a-9e07-48a6-861f-d38ffe1bd578/volumes/kubernetes.io~projected/kube-api-access-zcvhq:{mountpoint:/var/lib/kubelet/pods/8a79724a-9e07-48a6-861f-d38ffe1bd578/volumes/kubernetes.io~projected/kube-api-access-zcvhq major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~projected/kube-api-access-f7rgq:{mountpoint:/var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~projected/kube-api-access-f7rgq major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93773016-1fde-4819-905c-a0b774ab1b97/volumes/kubernetes.io~projected/kube-api-access-fp8ms:{mountpoint:/var/lib/kubelet/pods/93773016-1fde-4819-905c-a0b774ab1b97/volumes/kubernetes.io~projected/kube-api-access-fp8ms major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d3765b6-e817-4d17-8397-197521d8c888/volumes/kubernetes.io~projected/kube-api-access-vbzwm:{mountpoint:/var/lib/kubelet/pods/9d3765b6-e817-4d17-8397-197521d8c888/volumes/kubernetes.io~projected/kube-api-access-vbzwm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~projected/kube-api-access-fkjck:{mountpoint:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~projected/kube-api-access-fkjck major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~projected/kube-api-access-ff9k4:{mountpoint:/var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~projected/kube-api-access-ff9k4 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbaf9947-5532-47ce-aea5-dcb456c37684/volumes/kubernetes.io~projected/kube-api-access-gjhkh:{mountpoint:/var/lib/kubelet/pods/bbaf9947-5532-47ce-aea5-dcb456c37684/volumes/kubernetes.io~projected/kube-api-access-gjhkh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcf81c88-4fb2-4655-bd29-f2114fad2d3a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bcf81c88-4fb2-4655-bd29-f2114fad2d3a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/kube-api-access-5wzsf:{mountpoint:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/kube-api-access-5wzsf major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce27bdc4-89f2-4eed-a76d-57280fcb0daa/volumes/kubernetes.io~projected/kube-api-access-hk9zw:{mountpoint:/var/lib/kubelet/pods/ce27bdc4-89f2-4eed-a76d-57280fcb0daa/volumes/kubernetes.io~projected/kube-api-access-hk9zw major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd538562-960c-4a89-843b-fc901b120aa6/volumes/kubernetes.io~projected/kube-api-access-9lmdh:{mountpoint:/var/lib/kubelet/pods/dd538562-960c-4a89-843b-fc901b120aa6/volumes/kubernetes.io~projected/kube-api-access-9lmdh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0743f86-5e12-4a1d-baf8-fb1973fe4034/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e0743f86-5e12-4a1d-baf8-fb1973fe4034/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~projected/kube-api-access-fr942:{mountpoint:/var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~projected/kube-api-access-fr942 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~projected/kube-api-access-fbpll:{mountpoint:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~projected/kube-api-access-fbpll major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6975d49-682f-42d6-9ff2-21b90112b948/volumes/kubernetes.io~projected/kube-api-access-bxzpt:{mountpoint:/var/lib/kubelet/pods/e6975d49-682f-42d6-9ff2-21b90112b948/volumes/kubernetes.io~projected/kube-api-access-bxzpt major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef12024a-04a5-41ff-8bfc-18ddd645d296/volumes/kubernetes.io~projected/kube-api-access-2wgqr:{mountpoint:/var/lib/kubelet/pods/ef12024a-04a5-41ff-8bfc-18ddd645d296/volumes/kubernetes.io~projected/kube-api-access-2wgqr major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5faa68a-1aba-434f-94e8-83bdeedb2f28/volumes/kubernetes.io~projected/kube-api-access-49zk2:{mountpoint:/var/lib/kubelet/pods/f5faa68a-1aba-434f-94e8-83bdeedb2f28/volumes/kubernetes.io~projected/kube-api-access-49zk2 major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8e96383e35602f611b1b857c116efe04e7f318f46c44a0847d8b11bc52d23555/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b13f1dac871b5a4ff928867823707d85ab44370127b649337859aac7ed3c9830/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/30d09a634962c4607f0326db554369aa73f742c60f9ec940e917ef3d6ebc3122/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e14082d8e22458fc0325a233c6a8f5a257636d7ea5578a34aeca9a2209890a40/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7e04e2026fba0c116819a2908011c8d17604c36a38a544f20167e9d7da55945e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2eb5dbd7b654968ff5bf621510884ba1aad4d244cd12560fd3f8cd8c93cad6f6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c2468ee3d656ecc457721ef2c60190e60bc810486037ad2cb318488da5dd0020/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6b2f16965b529994c8b4475da7b13194c9955e8efa1b3efea99965fc013a75f2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1d5804bc1cfe2fc0f60472c1e80796895992ea7f42bb27f7de51a93fdbddf39b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e55d166f8999d58b29113ee00b8b38ebe321718f10b36898dc5c6f2602fe0b66/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/d1270b83e60e66391531c86b7b4ffa97f3dc2bd7180a4b1ddd9e84077ce0c5b7/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8631d7cea3b0039005f92e6521f4ae13dbaa0067848162e4b15e2da47de65757/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d30d3ce558b166f4e5bbaa58a3305d0ac662766f6499158963a0c9206c700912/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4269f1d41582d61402102c25423553fc9a2c38ce561d2e24016de9a366447e98/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/da133bd58c71b03446350c04e4e888b06e5805682fb4599673e1faa4f6b80594/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6747bc34e1d480d65ad86bef0cb8e7997fb9d7882bbd29bf0ebb2fad00133722/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b1dd146a5690ba5b6f6b0fa68aa81fb568e495b1540a372f64345a41893177e9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a5ffc7ee8ebaa20652d964ee85de5d4439373e3b759cbdd1ce3ff32531517dae/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ac2a61339c6c2111e980e5fa5f2f3f2544cc5d3be3a8da202ae06b9fb1b3e408/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4372d540e9f7f8624ae26b2b6b83fe8566d405cf04deffaf6ff2254ed37ae70f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/095c9169f9be6499fea19a2d6fb0d112ff0178ec5b9a25067fb992186cfb4b91/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3f84bc434bb3b740f97fb034dea1cd1ff8350355bef23a9a7aefe4239682c3f7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c108dd4fe571f505cb8dc88b88a7c8740584ea336880b0faac8303ac86a3b65f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/920123b9251103a4ea7d9a63adc52c90d4d3418e90aec6610b0338fb5c47c986/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/335729b5dc3f7372f1fb1db6ea4625ad3cab9bd6897ed575adf11314bb90b0e8/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/49a9309d4c9282090457c952765058eb2db45f49fb6bd5de5bbd48d3725e0e18/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d0348a0cab7f13f5b8c5404d215b440041e4fae2c246f562ab885af11a52de37/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d91b73d4f8b4f08cb17f19ed2e69852d6e52af4890d1f4c78a784fcde5528044/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d58cf62f01692f56b35708b0481434796d9387dd3ac0a9cfbf9cc622d6199e1b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/cbbce975ee3eb094c0973b83b435e2483b936a5315e20d38593fd8f2eca21ad3/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/8eddc63e400d97785c89729a5f5af4a9d7038b85a723a5824c982efb534af690/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/5cd743af7a49781758df46c5cf84b7ff7562dfc0cdc77e8bc4b3e0c8bffc76b0/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/0f0bd1f9258316d35ceecaf74c07fcd0c52929a07ca4bc46a431129ad8e12db1/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/aef4a5e6b172c02c7636d6603613c13814c96bba9f7ad23d08aaeb27a3b0d46a/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/53e7ae820c7ecce47badc288a07bb925a747e8345b04d69b23bec43cdb6f0c5d/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ff6250405c4eb73676f036bccf2b2f52f11555b2c409064b2be45b7582761431/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/704f7e362752feef57f86254d6252bc8de22776c62869678f3a8609b7fb414ba/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/bb8be6186ec34907fc8f10ef82c4455026dca05d496dc11a444a98151ac884d0/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ae8277c663620b382ad7ff3631be0d06f9d1ad77c973625576edff16a9842836/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b39c920ccc1cf4498bee5019304fe7fae3b5651205124618d711d482957e6d03/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/fb2f44661675b14f98169f2877ab9653577239ef3bc84a256aa9a4b2ef8f60d5/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a9fd10fe961767dd5196a9b9c508a0e7c4a41795808cf7bca6028bf90e520676/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/ebd4215f4e391aac9fc7c7c9f2eb27bc942f56a574a88fba5db680d0bce31e54/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/ce7795e106d41dcbb5255892d0fc3f2fe0aab9be661377da23efa9bd2dfec90a/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/98855c28b4d52818325348832e95d0d84316a8f8ff6f99e3319e176a34fd273f/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/694b03aa24ca855da8e235363ca048e7768d7e9cc309a4efa3f846268ba1fd3c/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/ea2f4931b2f7fd6426cc3f9e80a26a944caf454be85d5b4451fc3b228a89d25c/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/7051abcb00c7485c33c7ab717e7c556fa59da67b3c960626c7b1e81b4c6fbc2a/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/c6ce5b247e8e0b3bdb43b99ba13ea5038f57df04c1135a00863558a8e24f037d/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/dd31a8f0054f0c70acda1d319d932e38ebbf8b6ba3190e3093c5f7108397b4f2/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/de9aa8f3d1f3c915d3b9206a590ca9f37c0acef4b8c77d72e2a4140dc3c84636/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/5fccde4f2da28845a741fa8350cf0725f0ffb1cb8bc77c5815b79b0fc8f4aa3b/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/457924aa92c6e854b9f6beec5d9f3c261a9046842604e95ab72a103504ea396b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/181743be96f3e7f2454f5488790172fcc7dd41f81330f7f3a61f1f2385c4daad/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/451a215b5c19e47364643365cf4f86255a7fb80e39ed4075e898339a49aa104d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ee1d9f0616766a3edd05943805b453255cce4062d73f684696bc9f3e0f7a1559/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f935c1c296f1b81bbb8c77136ee70adb77e6b78322380c8b9ca131ff52dbdd4f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d370ba41aec4aac7c36f04fef798609683713e9dbb0ec76351eaf2cf3a54441f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/d58164b9db2f26be3cc5f091450e8f7fe91fb7a2a5dd79b828a9ce9b072d4f88/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5bed262a2a05e6c4d93fcaed30c42e4857e7294713d2d5bd30f45039ee3e0eb9/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a70656a4b90eb150820620a9a77cd00ee0c2962db75e4ca132fbae5fe284fdbd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cf5425db5a1a56c26352755a0286c2a9c664c202357a8de694250af915262d94/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 04:43:47.112466 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112103 5058 manager.go:217] Machine: {Timestamp:2026-10-30 04:43:47.111647484 +0000 UTC m=+0.081142465 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099955 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229f015a6493798740e583ba65ec2d SystemUUID:ec229f01-5a64-9379-8740-e583ba65ec2d BootID:d601d26e-98b9-4aea-be99-552f9dc20885 Filesystems:[{Device:/var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e6975d49-682f-42d6-9ff2-21b90112b948/volumes/kubernetes.io~projected/kube-api-access-bxzpt DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~projected/kube-api-access-fbpll DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18eb5265e3a3bee449e73e0e242be541823078c63337dd8df8efb9482df072b3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88/volumes/kubernetes.io~projected/kube-api-access-b99km DeviceMajor:0 DeviceMinor:333 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/947d19335970bfb0c49ade5cb7d5d0de070225e8e29dc6a9cf1aaef7d14eb37c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a560d198fc2ff799396429a7c21e78ab1099bd77bc7234fca83c90079476df7/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a79724a-9e07-48a6-861f-d38ffe1bd578/volumes/kubernetes.io~projected/kube-api-access-zcvhq DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/09e584bc-955f-4e38-a161-0258bdb03d24/volumes/kubernetes.io~projected/kube-api-access-qspx8 DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/kube-api-access-5wzsf DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c4807841770c685cb89fef734899d226e5b85b1d3d5d2584874e2dfd318d17b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de14bffd623eda664afde1b9b7f19acbb85af03cc6b64c0ceeef0c46e7e7d85a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5771f103-055c-43b2-b344-fb3f64cc0998/volumes/kubernetes.io~projected/kube-api-access-464tl DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~projected/kube-api-access-fkjck DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92/volumes/kubernetes.io~projected/kube-api-access-hq48z DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c95481855717e97819f0596f3e074810862aabe982220f7e9b8a631ff5f0aa9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/440d8e0c974c11eeb6ffd2e5440ac231a699345b754f9a249c874bc4c08bfef2/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/kube-api-access-79bn8 DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d84d597-b1ed-41d3-a4a2-1add913c1125/volumes/kubernetes.io~projected/kube-api-access-f7rgq DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16a3f8c9-121b-49f0-a822-12a395e21af7/volumes/kubernetes.io~projected/kube-api-access-zj2fv DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e/volumes/kubernetes.io~projected/kube-api-access-99ff4 DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0743f86-5e12-4a1d-baf8-fb1973fe4034/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0baadef4e29f5ecd01c5556883c58d9c40419e773263089d17db5c1d76c73cfe/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/984b208dbfd4271b46c60da19000bc655682bf0bfe757f166ae2081385e72877/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d3765b6-e817-4d17-8397-197521d8c888/volumes/kubernetes.io~projected/kube-api-access-vbzwm DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d72183b590c344eb11c2cf3b9c754b8651eb95e9ed70e1141bc98eac1d4257/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc/volumes/kubernetes.io~projected/kube-api-access-x5l2f DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e80f51467533c1df2201a83a1f5f691107151f5b3b486f61d6f7e3f277d0bed/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ade3d377dc4f9b276aa096357509b8687089c4da5323e568bd0fa9d662e09871/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b32ff5-8157-470f-94f6-ce77e4bf3898/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f5faa68a-1aba-434f-94e8-83bdeedb2f28/volumes/kubernetes.io~projected/kube-api-access-49zk2 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d022eb-a646-4f3b-8037-4f992cab9c21/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37948e1b-b416-4df5-9c56-7ff2e7996e48/volumes/kubernetes.io~projected/kube-api-access-5tjz5 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93773016-1fde-4819-905c-a0b774ab1b97/volumes/kubernetes.io~projected/kube-api-access-fp8ms DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28c20d88cc42a811fe323eaf07ab695070ea35052412fbb2d5a8b4220e028666/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dd538562-960c-4a89-843b-fc901b120aa6/volumes/kubernetes.io~projected/kube-api-access-9lmdh DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed91caeda97f8aceadf0c48345514dd4ab11d2c39e268d8e46d9c15c7648a85b/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~projected/kube-api-access-bzhsx DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80515748-a212-4b00-bec6-9b459babaa08/volumes/kubernetes.io~projected/kube-api-access-rj76p DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d46b9d0779ac39d19cacc1cd49b5e3f6297743bd5d61185958e0a730d395c58/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9de0f7f32653fa3bf063490ff3cbbc4854923ebf00fbb3578fa924d3d5d02e1/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df37a35df80abf8147ebdcf1cd65754a9a46a3df9b0b38f70396ca62279f5681/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbaf9947-5532-47ce-aea5-dcb456c37684/volumes/kubernetes.io~projected/kube-api-access-gjhkh DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bcf81c88-4fb2-4655-bd29-f2114fad2d3a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/3d95e075-6720-45ab-ab71-f8b5bc189697/volumes/kubernetes.io~projected/kube-api-access-5l9z2 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/48383d31-9302-4010-867c-0162421153e4/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a93c592f-d506-49e4-ba78-bdaa2de3561a/volumes/kubernetes.io~projected/kube-api-access-ff9k4 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef12024a-04a5-41ff-8bfc-18ddd645d296/volumes/kubernetes.io~projected/kube-api-access-2wgqr DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/139e01befa664ba835a7cacdf3df1be23f3f4858b9c6458770ce63d1fd66105f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2781cacad4c1cd71f09497400016c5aacf4cd76b768c86c5c4360018c9249b93/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce27bdc4-89f2-4eed-a76d-57280fcb0daa/volumes/kubernetes.io~projected/kube-api-access-hk9zw DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/115a2fb1-3568-46e3-90a2-98a88c00c27c/volumes/kubernetes.io~projected/kube-api-access-j477b DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2085ab02cb4d3bc178e5c311d81271c36af780866a14595f410f7d65407be0e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e11599cb-3adb-4c57-93e5-6a13d5fd50a1/volumes/kubernetes.io~projected/kube-api-access-fr942 DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64040d3f-1de7-42d3-a9aa-6fefa2a926ed/volumes/kubernetes.io~projected/kube-api-access-6dctq DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/801eb2b7-bb95-4b2f-95d6-44880fdf062c/volumes/kubernetes.io~projected/kube-api-access-npbhw DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4476bcec1ddc8ab97372921fe20f020ddc9a1a281fdfe2025b0aca965ebf1bc7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6264470d9075a18336fb6dda973d70d7aab68b79052702bb0c5cfdcd59c174a1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a8a0d3b37feb4b7af0c7e3e343c855c455879b034693de777d8a228e4ef400a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5183f685-7bd0-41b0-a632-9d6e56bc3951/volumes/kubernetes.io~projected/kube-api-access-lfdxk DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0baadef4e29f5ec MacAddress:62:9b:01:e4:fc:6f Speed:10000 Mtu:8901} {Name:23d72183b590c34 MacAddress:0a:fc:2d:36:49:5f Speed:10000 Mtu:8901} {Name:2781cacad4c1cd7 MacAddress:de:bc:94:e4:30:c8 Speed:10000 Mtu:8901} {Name:28c20d88cc42a81 MacAddress:a2:c8:e1:e6:db:fc Speed:10000 Mtu:8901} {Name:2e80f51467533c1 MacAddress:06:e9:ad:68:32:1f Speed:10000 Mtu:8901} {Name:3c95481855717e9 MacAddress:de:3d:12:29:3b:3a Speed:10000 Mtu:8901} {Name:440d8e0c974c11e MacAddress:6e:3d:fb:e0:f2:0d Speed:10000 Mtu:8901} {Name:6264470d9075a18 MacAddress:fa:1e:fd:c5:d4:6d Speed:10000 Mtu:8901} {Name:7a560d198fc2ff7 MacAddress:ae:18:32:8a:c3:a3 Speed:10000 Mtu:8901} {Name:7d46b9d0779ac39 MacAddress:fa:ed:b4:eb:e0:38 Speed:10000 Mtu:8901} {Name:984b208dbfd4271 MacAddress:fa:b5:a5:87:5f:ca Speed:10000 Mtu:8901} {Name:9cc93acf470b3d5 MacAddress:b2:44:a8:20:95:a2 Speed:10000 Mtu:8901} {Name:ade3d377dc4f9b2 MacAddress:4a:4d:f4:39:dc:8e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:eb:36:19 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:95:3f:5c:6d:db Speed:0 Mtu:8901} {Name:d9de0f7f32653fa MacAddress:96:8f:5e:b1:ab:6a Speed:10000 Mtu:8901} {Name:de14bffd623eda6 MacAddress:be:b8:65:3c:bd:22 Speed:10000 Mtu:8901} {Name:df37a35df80abf8 MacAddress:e6:d8:15:6b:ec:cd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:eb:36:19 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:79:75:4c:1e:e9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:36:7c:4c:c4:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 04:43:47.112466 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112467 5058 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 04:43:47.112780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112513 5058 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 04:43:47.112868 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112837 5058 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 04:43:47.112994 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112870 5058 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 04:43:47.113038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.112997 5058 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 04:43:47.113038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113005 5058 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 04:43:47.113038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113015 5058 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:47.113038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113035 5058 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:47.113143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113119 5058 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 04:43:47.113310 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113300 5058 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 04:43:47.113431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113422 5058 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 04:43:47.113475 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113467 5058 kubelet.go:493] "Attempting to sync node with API server" Oct 30 04:43:47.113511 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113477 5058 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 04:43:47.113511 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113487 5058 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 04:43:47.113511 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113496 5058 kubelet.go:405] "Adding apiserver pod source" Oct 30 04:43:47.113511 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.113504 5058 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 04:43:47.114334 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.114316 5058 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 04:43:47.114334 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.114334 5058 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:47.114871 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.114839 5058 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 04:43:47.117045 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117031 5058 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 04:43:47.117045 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117045 5058 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:47.117148 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117137 5058 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 04:43:47.117388 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117375 5058 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 04:43:47.117428 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117405 5058 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 04:43:47.117517 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117508 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117523 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117530 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117536 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117545 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117554 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 04:43:47.117564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117561 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117567 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117575 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117580 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117590 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117600 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117625 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 04:43:47.117807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.117632 5058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 04:43:47.121059 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.121044 5058 server.go:1263] "Started kubelet" Oct 30 04:43:47.121152 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.121108 5058 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 04:43:47.121233 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.121141 5058 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 04:43:47.121303 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.121265 5058 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 04:43:47.121603 ip-10-0-139-228 systemd[1]: Started Kubernetes Kubelet. Oct 30 04:43:47.121949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.121733 5058 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 04:43:47.123032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.122925 5058 server.go:317] "Adding debug handlers to kubelet server" Oct 30 04:43:47.128610 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.128413 5058 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 04:43:47.128778 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.128759 5058 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 04:43:47.128865 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.128837 5058 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:08 +0000 UTC" deadline="2028-04-29 21:25:37.041926739 +0000 UTC" Oct 30 04:43:47.128865 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.128861 5058 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13144h41m49.913069259s" Oct 30 04:43:47.129088 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.129072 5058 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 04:43:47.129149 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.129088 5058 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 04:43:47.129205 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.129193 5058 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 04:43:47.131445 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.131422 5058 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-228.ec2.internal\" not found" Oct 30 04:43:47.132850 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132812 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e584bc-955f-4e38-a161-0258bdb03d24" volumeName="kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume" seLinuxMountContext="" Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132851 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc" volumeName="kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f" seLinuxMountContext="" Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132867 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80515748-a212-4b00-bec6-9b459babaa08" volumeName="kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert" seLinuxMountContext="" Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132879 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d3765b6-e817-4d17-8397-197521d8c888" volumeName="kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm" seLinuxMountContext="" Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132886 5058 factory.go:55] Registering systemd factory Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132902 5058 factory.go:223] Registration of the systemd container factory successfully Oct 30 04:43:47.132939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132890 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbaf9947-5532-47ce-aea5-dcb456c37684" volumeName="kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132952 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6975d49-682f-42d6-9ff2-21b90112b948" volumeName="kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132967 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16a3f8c9-121b-49f0-a822-12a395e21af7" volumeName="kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132977 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" volumeName="kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132987 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d95e075-6720-45ab-ab71-f8b5bc189697" volumeName="kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.132995 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d2e0ccd-b10f-43ed-a75d-3672190d3d6e" volumeName="kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133003 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133012 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80515748-a212-4b00-bec6-9b459babaa08" volumeName="kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133019 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133027 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd538562-960c-4a89-843b-fc901b120aa6" volumeName="kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133041 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" volumeName="kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133053 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64040d3f-1de7-42d3-a9aa-6fefa2a926ed" volumeName="kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133065 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d022eb-a646-4f3b-8037-4f992cab9c21" volumeName="kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133078 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d022eb-a646-4f3b-8037-4f992cab9c21" volumeName="kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133089 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbaf9947-5532-47ce-aea5-dcb456c37684" volumeName="kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133144 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd538562-960c-4a89-843b-fc901b120aa6" volumeName="kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133160 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e11599cb-3adb-4c57-93e5-6a13d5fd50a1" volumeName="kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133172 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b32ff5-8157-470f-94f6-ce77e4bf3898" volumeName="kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133183 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" volumeName="kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133196 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37948e1b-b416-4df5-9c56-7ff2e7996e48" volumeName="kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133208 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64040d3f-1de7-42d3-a9aa-6fefa2a926ed" volumeName="kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133234 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133249 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133268 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133279 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b32ff5-8157-470f-94f6-ce77e4bf3898" volumeName="kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133288 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5faa68a-1aba-434f-94e8-83bdeedb2f28" volumeName="kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script" seLinuxMountContext="" Oct 30 04:43:47.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133298 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d95e075-6720-45ab-ab71-f8b5bc189697" volumeName="kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133310 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133323 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd538562-960c-4a89-843b-fc901b120aa6" volumeName="kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133333 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b32ff5-8157-470f-94f6-ce77e4bf3898" volumeName="kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133347 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e584bc-955f-4e38-a161-0258bdb03d24" volumeName="kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133362 5058 factory.go:153] Registering CRI-O factory Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133380 5058 factory.go:223] Registration of the crio container factory successfully Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133365 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc" volumeName="kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133447 5058 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133456 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48383d31-9302-4010-867c-0162421153e4" volumeName="kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133470 5058 factory.go:103] Registering Raw factory Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133470 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a79724a-9e07-48a6-861f-d38ffe1bd578" volumeName="kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133483 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93773016-1fde-4819-905c-a0b774ab1b97" volumeName="kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133497 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b32ff5-8157-470f-94f6-ce77e4bf3898" volumeName="kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133510 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48383d31-9302-4010-867c-0162421153e4" volumeName="kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133524 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d2e0ccd-b10f-43ed-a75d-3672190d3d6e" volumeName="kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133573 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf81c88-4fb2-4655-bd29-f2114fad2d3a" volumeName="kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133586 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="115a2fb1-3568-46e3-90a2-98a88c00c27c" volumeName="kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133597 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80515748-a212-4b00-bec6-9b459babaa08" volumeName="kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133610 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d022eb-a646-4f3b-8037-4f992cab9c21" volumeName="kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133623 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a93c592f-d506-49e4-ba78-bdaa2de3561a" volumeName="kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133648 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a93c592f-d506-49e4-ba78-bdaa2de3561a" volumeName="kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133671 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0743f86-5e12-4a1d-baf8-fb1973fe4034" volumeName="kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133683 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dae5dcf-7d46-4eba-a2c8-1193789e3c28" volumeName="kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133693 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d2e0ccd-b10f-43ed-a75d-3672190d3d6e" volumeName="kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133702 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133709 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133718 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133726 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133755 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd538562-960c-4a89-843b-fc901b120aa6" volumeName="kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133772 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b32ff5-8157-470f-94f6-ce77e4bf3898" volumeName="kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133780 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" volumeName="kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133788 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e11599cb-3adb-4c57-93e5-6a13d5fd50a1" volumeName="kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133799 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37948e1b-b416-4df5-9c56-7ff2e7996e48" volumeName="kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133806 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133813 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133821 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133828 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133838 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0743f86-5e12-4a1d-baf8-fb1973fe4034" volumeName="kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133847 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef12024a-04a5-41ff-8bfc-18ddd645d296" volumeName="kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133855 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d95e075-6720-45ab-ab71-f8b5bc189697" volumeName="kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133863 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48383d31-9302-4010-867c-0162421153e4" volumeName="kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133872 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5183f685-7bd0-41b0-a632-9d6e56bc3951" volumeName="kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133879 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133886 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fbbb93-c186-42f6-ba44-fd5b021e5d82" volumeName="kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133895 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801eb2b7-bb95-4b2f-95d6-44880fdf062c" volumeName="kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133903 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5faa68a-1aba-434f-94e8-83bdeedb2f28" volumeName="kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133911 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92" volumeName="kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133919 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801eb2b7-bb95-4b2f-95d6-44880fdf062c" volumeName="kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133927 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801eb2b7-bb95-4b2f-95d6-44880fdf062c" volumeName="kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133935 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133943 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133950 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd92781-ec40-4093-b4f3-5b91519a86b4" volumeName="kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133958 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e584bc-955f-4e38-a161-0258bdb03d24" volumeName="kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133965 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80515748-a212-4b00-bec6-9b459babaa08" volumeName="kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133973 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce27bdc4-89f2-4eed-a76d-57280fcb0daa" volumeName="kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133987 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5771f103-055c-43b2-b344-fb3f64cc0998" volumeName="kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133996 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64040d3f-1de7-42d3-a9aa-6fefa2a926ed" volumeName="kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134004 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a79724a-9e07-48a6-861f-d38ffe1bd578" volumeName="kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134011 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d84d597-b1ed-41d3-a4a2-1add913c1125" volumeName="kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134019 5058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e11599cb-3adb-4c57-93e5-6a13d5fd50a1" volumeName="kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert" seLinuxMountContext="" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134026 5058 reconstruct.go:97] "Volume reconstruction finished" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134031 5058 reconciler.go:29] "Reconciler: start to sync state" Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.133488 5058 manager.go:1199] Started watching for new ooms in manager Oct 30 04:43:47.134520 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.134521 5058 manager.go:319] Starting recovery of all containers Oct 30 04:43:47.151152 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.151084 5058 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 04:43:47.152636 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.152614 5058 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 04:43:47.152770 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.152640 5058 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 04:43:47.152770 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.152665 5058 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 04:43:47.152770 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.152715 5058 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 04:43:47.158307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.158212 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:47.158470 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.158442 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:47.168169 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.168119 5058 generic.go:356] "Generic (PLEG): container finished" podID="af1c879f72343f788edee4e1a247e807" containerID="bd163a301784c713afa0ece382b5c208a2b379efffab2584a9cc0b33ae99ef6f" exitCode=0 Oct 30 04:43:47.177538 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.177512 5058 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:47.177638 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.177572 5058 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 04:43:47.182044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.182026 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:47.182138 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.182055 5058 generic.go:356] "Generic (PLEG): container finished" podID="80515748-a212-4b00-bec6-9b459babaa08" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" exitCode=255 Oct 30 04:43:47.189761 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.189737 5058 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:47.191901 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.191877 5058 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:47.191983 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.191925 5058 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:47.207858 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207831 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="eaa435e9d13829d61da1b201fffaf1bc938224a41b2615527f1115f511c694fc" exitCode=0 Oct 30 04:43:47.207858 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207857 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="910476787ae2184d18bdbe760750a919940af30a12ff85657888e764ad5a818b" exitCode=0 Oct 30 04:43:47.208022 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207867 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="0c31af2eb69b3a5c007f557b6deae1925acf0845e9700492a7f273fce3b275e4" exitCode=0 Oct 30 04:43:47.208022 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207876 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="1f75fca03ac329e5348cf4aefac2187054f5a9f753e2e1845a01a6525267c26b" exitCode=0 Oct 30 04:43:47.208022 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207883 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="9850f490eb3c957e3ef9734397ae7c0bdc65b305c8202a346909aa4ecd4059be" exitCode=0 Oct 30 04:43:47.208022 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.207891 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd538562-960c-4a89-843b-fc901b120aa6" containerID="57326eda45dc90a280878c75d4b91d7f8a989023ad6dce691e1baa756a2fee1f" exitCode=0 Oct 30 04:43:47.238591 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.238568 5058 manager.go:324] Recovery completed Oct 30 04:43:47.253828 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.253799 5058 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 04:43:47.267468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267445 5058 cpu_manager.go:226] "Starting" policy="none" Oct 30 04:43:47.267468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267465 5058 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 04:43:47.267631 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267490 5058 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 04:43:47.267670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267648 5058 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 04:43:47.267699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267663 5058 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 04:43:47.267699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267685 5058 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 04:43:47.267699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267693 5058 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 04:43:47.267780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267702 5058 policy_none.go:50] "Start" Oct 30 04:43:47.267780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267713 5058 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 04:43:47.267780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267723 5058 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 04:43:47.267863 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267851 5058 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 04:43:47.267892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267864 5058 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 04:43:47.267892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.267870 5058 policy_none.go:44] "Start" Oct 30 04:43:47.272417 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.272384 5058 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 04:43:47.275864 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.275848 5058 manager.go:341] "Starting Device Plugin manager" Oct 30 04:43:47.276605 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.276584 5058 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 04:43:47.276668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.276608 5058 server.go:87] "Starting device plugin registration server" Oct 30 04:43:47.276940 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.276924 5058 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 04:43:47.276991 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.276946 5058 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 04:43:47.277178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.277156 5058 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 04:43:47.277297 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.277270 5058 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 04:43:47.277352 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.277308 5058 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 04:43:47.277352 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.277315 5058 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 04:43:47.277472 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.277387 5058 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:43:47.277298152Z","UUID":"c2d7e60b-df36-4bba-9dfe-88b209b35b86","Handler":null,"Name":"","Endpoint":""} Oct 30 04:43:47.278717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.278698 5058 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 04:43:47.278811 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.278725 5058 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 04:43:47.278811 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.278774 5058 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 04:43:47.377972 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.377908 5058 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:47.379110 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.379093 5058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:47.379190 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.379127 5058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:47.379190 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.379142 5058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-228.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:47.379285 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.379198 5058 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.414741 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.414697 5058 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.414871 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.414789 5058 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.453978 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.453928 5058 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal"] Oct 30 04:43:47.454664 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454610 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" event={"ID":"cd885c44a562061a49f43990cf7668c0","Type":"ContainerStarted","Data":"0f2424c780cf6e8c5ad40749891457a85ed34a79f127990b247d3bafd2ee3364"} Oct 30 04:43:47.454737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454666 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" event={"ID":"cd885c44a562061a49f43990cf7668c0","Type":"ContainerStarted","Data":"9ae6cf0b39aff0df5030d7387c91683fa6e8796c11af263609eda494789ea4fb"} Oct 30 04:43:47.454737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454662 5058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.454737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454690 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerStarted","Data":"39bf65cf40e2e38e5c9cb675e3b51f7354d6963d95907304469beb3f655a7a22"} Oct 30 04:43:47.454737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454707 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerDied","Data":"bd163a301784c713afa0ece382b5c208a2b379efffab2584a9cc0b33ae99ef6f"} Oct 30 04:43:47.454737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454726 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" event={"ID":"af1c879f72343f788edee4e1a247e807","Type":"ContainerStarted","Data":"bae618777dd12dbb43d914ae1f4b8a752a10cc405883b973c9f7056fb45f0dd4"} Oct 30 04:43:47.454893 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.454877 5058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.479778 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.479751 5058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:47.479862 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.479800 5058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.480213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.480198 5058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:47.480278 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:47.480254 5058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.579668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.579634 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.579668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.579667 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.579859 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.579689 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680300 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680257 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680300 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680299 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680550 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680314 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680550 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680367 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680550 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680376 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af1c879f72343f788edee4e1a247e807-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" (UID: \"af1c879f72343f788edee4e1a247e807\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:47.680550 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:47.680430 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd885c44a562061a49f43990cf7668c0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" (UID: \"cd885c44a562061a49f43990cf7668c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:48.114055 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.113980 5058 apiserver.go:52] "Watching apiserver" Oct 30 04:43:48.171950 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.171917 5058 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 04:43:48.172684 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.172660 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg","kube-system/konnectivity-agent-nrrc2","openshift-console-operator/console-operator-7cd56b8dbc-wf2dd","openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q","kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal","openshift-insights/insights-runtime-extractor-zh2bq","openshift-dns/dns-default-hq5jn","openshift-multus/multus-additional-cni-plugins-nfrbv","kube-system/kas-connection-checker-685895c7c9-9ff65","openshift-network-diagnostics/network-check-target-6v95n","openshift-network-operator/iptables-alerter-577zz","kube-system/kas-connection-checker-685895c7c9-4j2vn","openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5","openshift-cluster-node-tuning-operator/tuned-s7nn8","openshift-insights/insights-operator-7c4986d449-d82h8","openshift-ovn-kubernetes/ovnkube-node-s445b","openshift-multus/multus-8krtz","openshift-dns/node-resolver-t9k5j","openshift-ingress/router-default-74b99df667-5m6dw","openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9","openshift-multus/network-metrics-daemon-2fszz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal","kube-system/kas-connection-checker-685895c7c9-cddr7","openshift-insights/periodic-gathering-kz2hv-m8wbg","openshift-network-console/networking-console-plugin-86c478bb88-84nvd","openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc","openshift-image-registry/node-ca-lphjh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l","openshift-ingress-canary/ingress-canary-79pwp","openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94","openshift-service-ca/service-ca-5fffc97bdf-nxbx2","openshift-image-registry/image-registry-7f6f8cd844-zgvtd","kube-system/global-pull-secret-syncer-7tbzb"] Oct 30 04:43:48.173880 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.173859 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.174879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.174859 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.175555 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.175535 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.176250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.176232 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.177257 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.177241 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.184651 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.184623 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.186256 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.186238 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.187297 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.187281 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.187813 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.187798 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.188692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.188674 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.188692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.188678 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.189751 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.189732 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.189831 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.189797 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 04:43:48.190032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190016 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.190162 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190108 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 04:43:48.190248 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190172 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 04:43:48.190248 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190183 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 04:43:48.190370 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190252 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 04:43:48.190370 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190252 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.190370 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190324 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.190674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190050 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 04:43:48.190674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190437 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 04:43:48.190674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190448 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 04:43:48.190674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190484 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.190674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.190639 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.203460 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.203440 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 04:43:48.203551 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.203538 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 04:43:48.203621 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.203603 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9lcst\"" Oct 30 04:43:48.203710 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.203696 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 04:43:48.204189 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204175 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 04:43:48.204278 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204211 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\"" Oct 30 04:43:48.204278 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204255 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 04:43:48.204369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204327 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 04:43:48.204369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204343 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\"" Oct 30 04:43:48.204369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204359 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 04:43:48.204489 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204416 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\"" Oct 30 04:43:48.204581 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204564 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\"" Oct 30 04:43:48.204635 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204580 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\"" Oct 30 04:43:48.204635 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204593 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 04:43:48.204635 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204593 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\"" Oct 30 04:43:48.204767 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204734 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\"" Oct 30 04:43:48.204767 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204744 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 04:43:48.204767 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204754 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204766 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204744 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204732 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204844 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204866 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmmt4\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204881 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 04:43:48.204905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204895 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204948 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.204998 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205040 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205076 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205081 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205102 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205163 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205170 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205185 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205233 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 04:43:48.205330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205280 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205354 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205359 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205398 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205358 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205400 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205419 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205456 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205477 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205483 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205517 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205516 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205528 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205538 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205574 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205578 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205600 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205615 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205621 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205638 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205654 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205656 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wfdkq\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205691 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205698 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205703 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205705 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205728 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205748 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205759 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 04:43:48.205805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205766 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205943 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205944 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205975 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205983 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205994 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.205977 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206047 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206074 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206091 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206134 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206190 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206383 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206419 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tqz9q\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206427 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206485 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206492 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206484 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206553 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206553 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 04:43:48.206890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.206664 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 04:43:48.216814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.216796 5058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:48.216922 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.216910 5058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:48.217766 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.217742 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 04:43:48.217975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.217939 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dz97m\"" Oct 30 04:43:48.218099 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.218078 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 04:43:48.223010 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.222981 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 04:43:48.223161 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.223143 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 04:43:48.231170 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.231153 5058 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 04:43:48.240108 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.240092 5058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:48.240167 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.240130 5058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-228.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-228.ec2.internal" Oct 30 04:43:48.243615 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.243597 5058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:48.243680 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.243646 5058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-228.ec2.internal" Oct 30 04:43:48.285043 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285017 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.285143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285050 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.285143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285078 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.285265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285143 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zk2\" (UniqueName: \"kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.285265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285167 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbhw\" (UniqueName: \"kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.285265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285185 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.285265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285230 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.285265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285253 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.285527 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285277 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.285527 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285327 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.285527 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285407 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/801eb2b7-bb95-4b2f-95d6-44880fdf062c-config-volume\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.285679 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285517 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.285679 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.285679 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285621 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.285679 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285635 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-cni-binary-copy\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.285679 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-service-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285680 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285708 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285735 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285792 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285743 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285820 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285830 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285836 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-key\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285847 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285859 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.285910 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285902 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.286234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285951 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.286234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.285986 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspx8\" (UniqueName: \"kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.286234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286015 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq48z\" (UniqueName: \"kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z\") pod \"kas-connection-checker-685895c7c9-cddr7\" (UID: \"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92\") " pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:48.286234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286107 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.286234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286115 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.286380 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286256 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.286380 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286279 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.286380 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286301 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:48.286380 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286335 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.286534 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286375 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2fv\" (UniqueName: \"kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv\") pod \"network-check-source-6f78d59f99-qxx47\" (UID: \"16a3f8c9-121b-49f0-a822-12a395e21af7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:48.286534 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286420 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvhq\" (UniqueName: \"kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.286534 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286450 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.286534 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286477 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.286534 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286507 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286537 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286563 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-serving-cert\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286563 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286595 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286600 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d84d597-b1ed-41d3-a4a2-1add913c1125-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286726 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286780 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.286815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286809 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286830 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286848 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286850 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-config\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286873 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj76p\" (UniqueName: \"kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286904 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr942\" (UniqueName: \"kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286930 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93c592f-d506-49e4-ba78-bdaa2de3561a-cert\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286952 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.286979 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287094 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287095 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-stats-auth\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287145 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287141 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-daemon-config\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.287178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287179 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287212 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287280 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287309 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287202 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-script-lib\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287377 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j477b\" (UniqueName: \"kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b\") pod \"migrator-66df5d496b-zt75v\" (UID: \"115a2fb1-3568-46e3-90a2-98a88c00c27c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287465 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287483 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287520 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovn-node-metrics-cert\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287628 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287703 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287728 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.287787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287752 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287786 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0743f86-5e12-4a1d-baf8-fb1973fe4034-konnectivity-ca\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287806 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287849 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287903 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ff4\" (UniqueName: \"kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287993 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.287994 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288096 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288171 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.288529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288274 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801eb2b7-bb95-4b2f-95d6-44880fdf062c-tmp-dir\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.288529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288288 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.288529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288344 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.288529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288418 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.288529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288487 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.288697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288560 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.288697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288599 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.288697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288646 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.288697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288682 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.288830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288721 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.288830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288731 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80515748-a212-4b00-bec6-9b459babaa08-trusted-ca\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.288830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288771 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.288830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288797 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464tl\" (UniqueName: \"kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.288830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288817 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.288971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288851 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.288971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288870 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.288971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288877 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgqr\" (UniqueName: \"kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr\") pod \"migrator-66df5d496b-kgcn9\" (UID: \"ef12024a-04a5-41ff-8bfc-18ddd645d296\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:48.288971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288911 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.289098 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288974 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.289098 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.288980 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.289098 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289008 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.289251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289105 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.289251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289229 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.289251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289238 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.289399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289256 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjck\" (UniqueName: \"kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.289399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289300 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.289399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289330 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9z2\" (UniqueName: \"kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.289399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289355 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.289524 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289424 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.289524 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289512 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.289606 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289587 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.289654 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289629 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.289699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289681 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjz5\" (UniqueName: \"kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.289757 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289725 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.289798 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289756 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-tmp\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.289798 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289776 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.289866 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289831 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.289866 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289851 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.289866 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289861 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dctq\" (UniqueName: \"kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.289996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289888 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.289996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289913 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsx\" (UniqueName: \"kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.289996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289930 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.289996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289890 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5faa68a-1aba-434f-94e8-83bdeedb2f28-iptables-alerter-script\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.289996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.289956 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290014 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-env-overrides\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290014 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpll\" (UniqueName: \"kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290054 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290081 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290092 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80515748-a212-4b00-bec6-9b459babaa08-serving-cert\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290107 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290133 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290148 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-snapshots\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290164 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290158 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09e584bc-955f-4e38-a161-0258bdb03d24-data-volume\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.290243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290191 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290360 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290374 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-signing-cabundle\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290437 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290484 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290515 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290541 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290566 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290596 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290607 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290611 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290624 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290655 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290753 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.290779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290784 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290800 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290819 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290836 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8ms\" (UniqueName: \"kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms\") pod \"kas-connection-checker-685895c7c9-9ff65\" (UID: \"93773016-1fde-4819-905c-a0b774ab1b97\") " pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290856 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290917 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-tuned\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.290972 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291052 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09e584bc-955f-4e38-a161-0258bdb03d24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291103 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291120 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291135 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9k4\" (UniqueName: \"kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291153 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291198 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdxk\" (UniqueName: \"kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk\") pod \"kas-connection-checker-685895c7c9-4j2vn\" (UID: \"5183f685-7bd0-41b0-a632-9d6e56bc3951\") " pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291227 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291239 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-default-certificate\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291251 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291268 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmdh\" (UniqueName: \"kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291284 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l2f\" (UniqueName: \"kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291300 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291316 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291198 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd538562-960c-4a89-843b-fc901b120aa6-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291333 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291379 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291397 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzpt\" (UniqueName: \"kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt\") pod \"volume-data-source-validator-58c9f5f46d-jxfs5\" (UID: \"e6975d49-682f-42d6-9ff2-21b90112b948\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291424 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291439 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.291440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291455 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291473 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291490 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291505 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291519 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291569 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rgq\" (UniqueName: \"kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291601 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291618 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhkh\" (UniqueName: \"kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291634 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291649 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291673 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291691 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291706 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291722 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291743 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291759 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291774 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.291791 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292074 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d84d597-b1ed-41d3-a4a2-1add913c1125-serving-cert\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292295 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbaf9947-5532-47ce-aea5-dcb456c37684-serviceca\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292298 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-nginx-conf\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292474 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37948e1b-b416-4df5-9c56-7ff2e7996e48-tmp-dir\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292595 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4b32ff5-8157-470f-94f6-ce77e4bf3898-ovnkube-config\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.292683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.292644 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d84d597-b1ed-41d3-a4a2-1add913c1125-tmp\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.293565 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.293043 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-original-pull-secret\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.293565 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.293058 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0743f86-5e12-4a1d-baf8-fb1973fe4034-agent-certs\") pod \"konnectivity-agent-nrrc2\" (UID: \"e0743f86-5e12-4a1d-baf8-fb1973fe4034\") " pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:43:48.293565 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.293089 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-config\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.305138 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.305115 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zk2\" (UniqueName: \"kubernetes.io/projected/f5faa68a-1aba-434f-94e8-83bdeedb2f28-kube-api-access-49zk2\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.305453 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.305424 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbhw\" (UniqueName: \"kubernetes.io/projected/801eb2b7-bb95-4b2f-95d6-44880fdf062c-kube-api-access-npbhw\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.305593 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.305571 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.305916 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.305894 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspx8\" (UniqueName: \"kubernetes.io/projected/09e584bc-955f-4e38-a161-0258bdb03d24-kube-api-access-qspx8\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.305969 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.305948 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq48z\" (UniqueName: \"kubernetes.io/projected/6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92-kube-api-access-hq48z\") pod \"kas-connection-checker-685895c7c9-cddr7\" (UID: \"6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92\") " pod="kube-system/kas-connection-checker-685895c7c9-cddr7" Oct 30 04:43:48.306078 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.306058 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9zw\" (UniqueName: \"kubernetes.io/projected/ce27bdc4-89f2-4eed-a76d-57280fcb0daa-kube-api-access-hk9zw\") pod \"network-check-target-6v95n\" (UID: \"ce27bdc4-89f2-4eed-a76d-57280fcb0daa\") " pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:48.306115 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.306068 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvhq\" (UniqueName: \"kubernetes.io/projected/8a79724a-9e07-48a6-861f-d38ffe1bd578-kube-api-access-zcvhq\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.306145 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.306121 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2fv\" (UniqueName: \"kubernetes.io/projected/16a3f8c9-121b-49f0-a822-12a395e21af7-kube-api-access-zj2fv\") pod \"network-check-source-6f78d59f99-qxx47\" (UID: \"16a3f8c9-121b-49f0-a822-12a395e21af7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qxx47" Oct 30 04:43:48.306450 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.306434 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr942\" (UniqueName: \"kubernetes.io/projected/e11599cb-3adb-4c57-93e5-6a13d5fd50a1-kube-api-access-fr942\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sfq94\" (UID: \"e11599cb-3adb-4c57-93e5-6a13d5fd50a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" Oct 30 04:43:48.306699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.306684 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj76p\" (UniqueName: \"kubernetes.io/projected/80515748-a212-4b00-bec6-9b459babaa08-kube-api-access-rj76p\") pod \"console-operator-7cd56b8dbc-wf2dd\" (UID: \"80515748-a212-4b00-bec6-9b459babaa08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:48.311096 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311070 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464tl\" (UniqueName: \"kubernetes.io/projected/5771f103-055c-43b2-b344-fb3f64cc0998-kube-api-access-464tl\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.311096 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311087 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgqr\" (UniqueName: \"kubernetes.io/projected/ef12024a-04a5-41ff-8bfc-18ddd645d296-kube-api-access-2wgqr\") pod \"migrator-66df5d496b-kgcn9\" (UID: \"ef12024a-04a5-41ff-8bfc-18ddd645d296\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kgcn9" Oct 30 04:43:48.311480 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311461 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ff4\" (UniqueName: \"kubernetes.io/projected/6d2e0ccd-b10f-43ed-a75d-3672190d3d6e-kube-api-access-99ff4\") pod \"service-ca-5fffc97bdf-nxbx2\" (UID: \"6d2e0ccd-b10f-43ed-a75d-3672190d3d6e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nxbx2" Oct 30 04:43:48.311777 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311761 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.311815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311765 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"periodic-gathering-kz2hv-m8wbg\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:43:48.311815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.311800 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j477b\" (UniqueName: \"kubernetes.io/projected/115a2fb1-3568-46e3-90a2-98a88c00c27c-kube-api-access-j477b\") pod \"migrator-66df5d496b-zt75v\" (UID: \"115a2fb1-3568-46e3-90a2-98a88c00c27c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zt75v" Oct 30 04:43:48.314497 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.314475 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l2f\" (UniqueName: \"kubernetes.io/projected/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-kube-api-access-x5l2f\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.314990 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.314970 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.315614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.315526 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjck\" (UniqueName: \"kubernetes.io/projected/a0d022eb-a646-4f3b-8037-4f992cab9c21-kube-api-access-fkjck\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.315832 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.315809 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzpt\" (UniqueName: \"kubernetes.io/projected/e6975d49-682f-42d6-9ff2-21b90112b948-kube-api-access-bxzpt\") pod \"volume-data-source-validator-58c9f5f46d-jxfs5\" (UID: \"e6975d49-682f-42d6-9ff2-21b90112b948\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jxfs5" Oct 30 04:43:48.316038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316005 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dctq\" (UniqueName: \"kubernetes.io/projected/64040d3f-1de7-42d3-a9aa-6fefa2a926ed-kube-api-access-6dctq\") pod \"service-ca-operator-544d749c8c-krmmv\" (UID: \"64040d3f-1de7-42d3-a9aa-6fefa2a926ed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" Oct 30 04:43:48.316038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316027 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhsx\" (UniqueName: \"kubernetes.io/projected/48383d31-9302-4010-867c-0162421153e4-kube-api-access-bzhsx\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.316200 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316020 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdxk\" (UniqueName: \"kubernetes.io/projected/5183f685-7bd0-41b0-a632-9d6e56bc3951-kube-api-access-lfdxk\") pod \"kas-connection-checker-685895c7c9-4j2vn\" (UID: \"5183f685-7bd0-41b0-a632-9d6e56bc3951\") " pod="kube-system/kas-connection-checker-685895c7c9-4j2vn" Oct 30 04:43:48.316200 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316020 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8ms\" (UniqueName: \"kubernetes.io/projected/93773016-1fde-4819-905c-a0b774ab1b97-kube-api-access-fp8ms\") pod \"kas-connection-checker-685895c7c9-9ff65\" (UID: \"93773016-1fde-4819-905c-a0b774ab1b97\") " pod="kube-system/kas-connection-checker-685895c7c9-9ff65" Oct 30 04:43:48.316200 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316175 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjz5\" (UniqueName: \"kubernetes.io/projected/37948e1b-b416-4df5-9c56-7ff2e7996e48-kube-api-access-5tjz5\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.316630 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316608 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpll\" (UniqueName: \"kubernetes.io/projected/e4b32ff5-8157-470f-94f6-ce77e4bf3898-kube-api-access-fbpll\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.316692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316616 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9z2\" (UniqueName: \"kubernetes.io/projected/3d95e075-6720-45ab-ab71-f8b5bc189697-kube-api-access-5l9z2\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.316768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316748 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.316811 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316767 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/9d3765b6-e817-4d17-8397-197521d8c888-kube-api-access-vbzwm\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.316876 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316858 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9k4\" (UniqueName: \"kubernetes.io/projected/a93c592f-d506-49e4-ba78-bdaa2de3561a-kube-api-access-ff9k4\") pod \"ingress-canary-79pwp\" (UID: \"a93c592f-d506-49e4-ba78-bdaa2de3561a\") " pod="openshift-ingress-canary/ingress-canary-79pwp" Oct 30 04:43:48.316945 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.316925 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rgq\" (UniqueName: \"kubernetes.io/projected/8d84d597-b1ed-41d3-a4a2-1add913c1125-kube-api-access-f7rgq\") pod \"insights-operator-7c4986d449-d82h8\" (UID: \"8d84d597-b1ed-41d3-a4a2-1add913c1125\") " pod="openshift-insights/insights-operator-7c4986d449-d82h8" Oct 30 04:43:48.317015 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.317001 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhkh\" (UniqueName: \"kubernetes.io/projected/bbaf9947-5532-47ce-aea5-dcb456c37684-kube-api-access-gjhkh\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.317054 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.317040 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmdh\" (UniqueName: \"kubernetes.io/projected/dd538562-960c-4a89-843b-fc901b120aa6-kube-api-access-9lmdh\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.392301 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392270 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.392301 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392303 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392319 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392371 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-k8s-cni-cncf-io\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392397 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392412 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392426 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392444 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392452 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-hostroot\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392469 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392485 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-run\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392505 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392485 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-kubernetes\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392490 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-socket-dir-parent\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392521 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-kubelet\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392519 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbaf9947-5532-47ce-aea5-dcb456c37684-host\") pod \"node-ca-lphjh\" (UID: \"bbaf9947-5532-47ce-aea5-dcb456c37684\") " pod="openshift-image-registry/node-ca-lphjh" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392574 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392570 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-lib-modules\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392613 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392641 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392666 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392677 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-conf\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392674 5058 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392691 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-kubelet-config\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392693 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-bin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392691 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392717 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-cnibin\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392733 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392754 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392763 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-kubelet\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392777 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392796 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392801 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392831 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392833 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392849 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392868 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-var-lib-cni-multus\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392868 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392897 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-multus-conf-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392900 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392926 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392943 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392960 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.392962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392978 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-device-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392996 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-var-lib-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.392981 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393031 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-netd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393036 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-cni-bin\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.393042 5058 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393051 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393077 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393103 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.393122 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:48.893100564 +0000 UTC m=+1.862595550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393121 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393161 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf81c88-4fb2-4655-bd29-f2114fad2d3a-dbus\") pod \"global-pull-secret-syncer-7tbzb\" (UID: \"bcf81c88-4fb2-4655-bd29-f2114fad2d3a\") " pod="kube-system/global-pull-secret-syncer-7tbzb" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393185 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393213 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37948e1b-b416-4df5-9c56-7ff2e7996e48-hosts-file\") pod \"node-resolver-t9k5j\" (UID: \"37948e1b-b416-4df5-9c56-7ff2e7996e48\") " pod="openshift-dns/node-resolver-t9k5j" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393239 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393269 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393273 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd538562-960c-4a89-843b-fc901b120aa6-system-cni-dir\") pod \"multus-additional-cni-plugins-nfrbv\" (UID: \"dd538562-960c-4a89-843b-fc901b120aa6\") " pod="openshift-multus/multus-additional-cni-plugins-nfrbv" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393275 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-netns\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393368 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393412 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393438 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393471 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393493 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393527 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393535 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-modprobe-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393555 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393493 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-systemd\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393582 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393608 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393612 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-cnibin\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393634 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393677 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-socket-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393681 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393718 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393736 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-registration-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393747 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393771 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393773 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-netns\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393796 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393822 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393825 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393859 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-ovn\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393875 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-sys-fs\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393886 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393898 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-systemd-units\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393913 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-run-systemd\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393924 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393954 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysconfig\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393956 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393988 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.393999 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-log-socket\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394015 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394040 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394064 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394093 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394125 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394156 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394188 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394247 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394273 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394301 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394328 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394350 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394383 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09e584bc-955f-4e38-a161-0258bdb03d24-crio-socket\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394426 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-slash\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394433 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-host-run-multus-certs\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394470 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-etc-openvswitch\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.394539 5058 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394553 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-etc-sysctl-d\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.394597 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:48.894579867 +0000 UTC m=+1.864074847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : secret "metrics-daemon-secret" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.394641 5058 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-sys\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394683 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5faa68a-1aba-434f-94e8-83bdeedb2f28-host-slash\") pod \"iptables-alerter-577zz\" (UID: \"f5faa68a-1aba-434f-94e8-83bdeedb2f28\") " pod="openshift-network-operator/iptables-alerter-577zz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.394690 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:48.894667783 +0000 UTC m=+1.864162765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394732 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d3765b6-e817-4d17-8397-197521d8c888-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6b6l\" (UID: \"9d3765b6-e817-4d17-8397-197521d8c888\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6b6l" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394738 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-var-lib-kubelet\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.394783 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:48.89475918 +0000 UTC m=+1.864254170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394811 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d022eb-a646-4f3b-8037-4f992cab9c21-host\") pod \"tuned-s7nn8\" (UID: \"a0d022eb-a646-4f3b-8037-4f992cab9c21\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nn8" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394812 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-system-cni-dir\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394847 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-host-run-ovn-kubernetes\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394863 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4b32ff5-8157-470f-94f6-ce77e4bf3898-node-log\") pod \"ovnkube-node-s445b\" (UID: \"e4b32ff5-8157-470f-94f6-ce77e4bf3898\") " pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:48.395151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.394850 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d95e075-6720-45ab-ab71-f8b5bc189697-etc-kubernetes\") pod \"multus-8krtz\" (UID: \"3d95e075-6720-45ab-ab71-f8b5bc189697\") " pod="openshift-multus/multus-8krtz" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.395525 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"image-registry-5dfc9b8cc-wb6qg\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.395598 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"image-registry-7f6f8cd844-zgvtd\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.395857 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48383d31-9302-4010-867c-0162421153e4-metrics-certs\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.396196 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l2bhc\" (UID: \"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.396840 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dae5dcf-7d46-4eba-a2c8-1193789e3c28-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-84nvd\" (UID: \"5dae5dcf-7d46-4eba-a2c8-1193789e3c28\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.398415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.397529 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/801eb2b7-bb95-4b2f-95d6-44880fdf062c-metrics-tls\") pod \"dns-default-hq5jn\" (UID: \"801eb2b7-bb95-4b2f-95d6-44880fdf062c\") " pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.474794 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.474765 5058 scope.go:122] "RemoveContainer" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" Oct 30 04:43:48.475151 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.475132 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" Oct 30 04:43:48.475818 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.475798 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:48.477502 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.477485 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" Oct 30 04:43:48.486642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.486616 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:48.487796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.487776 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:48.807171 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.807139 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc"] Oct 30 04:43:48.808182 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:48.808145 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1c0f74_26d2_4294_aafe_1d8b22ed2fbc.slice/crio-977305a142df87be2d2b6d5959f5e0b8f6a5916123d56a139da72dd307080697 WatchSource:0}: Error finding container 977305a142df87be2d2b6d5959f5e0b8f6a5916123d56a139da72dd307080697: Status 404 returned error can't find the container with id 977305a142df87be2d2b6d5959f5e0b8f6a5916123d56a139da72dd307080697 Oct 30 04:43:48.810096 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.810079 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:43:48.850825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.850804 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:43:48.852570 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:48.852530 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd92781_ec40_4093_b4f3_5b91519a86b4.slice/crio-e5937e6b9c4bf0c76bbe1f2e2f7dfd84bd0ea7a64e240ebfd81832bc85af78df WatchSource:0}: Error finding container e5937e6b9c4bf0c76bbe1f2e2f7dfd84bd0ea7a64e240ebfd81832bc85af78df: Status 404 returned error can't find the container with id e5937e6b9c4bf0c76bbe1f2e2f7dfd84bd0ea7a64e240ebfd81832bc85af78df Oct 30 04:43:48.883731 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.883711 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hq5jn"] Oct 30 04:43:48.884831 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:48.884805 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801eb2b7_bb95_4b2f_95d6_44880fdf062c.slice/crio-d6b3d8fa9c46eb9b2e6ff1ce849145653e85316669998c61da7a4da661f836e0 WatchSource:0}: Error finding container d6b3d8fa9c46eb9b2e6ff1ce849145653e85316669998c61da7a4da661f836e0: Status 404 returned error can't find the container with id d6b3d8fa9c46eb9b2e6ff1ce849145653e85316669998c61da7a4da661f836e0 Oct 30 04:43:48.898208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.898161 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:48.898338 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.898235 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:48.898338 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.898257 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:48.898338 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.898280 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898376 5058 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898398 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:49.898379116 +0000 UTC m=+2.867874101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898401 5058 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898445 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:49.898427793 +0000 UTC m=+2.867922771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898454 5058 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898465 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs podName:5771f103-055c-43b2-b344-fb3f64cc0998 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:49.898455222 +0000 UTC m=+2.867950196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs") pod "network-metrics-daemon-2fszz" (UID: "5771f103-055c-43b2-b344-fb3f64cc0998") : secret "metrics-daemon-secret" not found Oct 30 04:43:48.898497 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:48.898500 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:49.898487779 +0000 UTC m=+2.867982774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:48.916764 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.916739 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-84nvd"] Oct 30 04:43:48.918168 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:48.918146 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dae5dcf_7d46_4eba_a2c8_1193789e3c28.slice/crio-2ca47d251b2daa514d9fe5f2856ca9569f165546e0a90027e474e1c5b21f1598 WatchSource:0}: Error finding container 2ca47d251b2daa514d9fe5f2856ca9569f165546e0a90027e474e1c5b21f1598: Status 404 returned error can't find the container with id 2ca47d251b2daa514d9fe5f2856ca9569f165546e0a90027e474e1c5b21f1598 Oct 30 04:43:48.948356 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:48.948329 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:43:48.950283 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:48.950258 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fbbb93_c186_42f6_ba44_fd5b021e5d82.slice/crio-623cd4aec0148bdeda24181aed98b172236929baad9b2d54c55a2489bad7d027 WatchSource:0}: Error finding container 623cd4aec0148bdeda24181aed98b172236929baad9b2d54c55a2489bad7d027: Status 404 returned error can't find the container with id 623cd4aec0148bdeda24181aed98b172236929baad9b2d54c55a2489bad7d027 Oct 30 04:43:49.227981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.227885 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" event={"ID":"ccd92781-ec40-4093-b4f3-5b91519a86b4","Type":"ContainerStarted","Data":"9e69c2af3cc937f1325decfa8dc43be2a9fa75267baf8e5041dff89e5241ddaa"} Oct 30 04:43:49.227981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.227974 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" event={"ID":"ccd92781-ec40-4093-b4f3-5b91519a86b4","Type":"ContainerStarted","Data":"e5937e6b9c4bf0c76bbe1f2e2f7dfd84bd0ea7a64e240ebfd81832bc85af78df"} Oct 30 04:43:49.229654 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.229630 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" event={"ID":"76fbbb93-c186-42f6-ba44-fd5b021e5d82","Type":"ContainerStarted","Data":"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c"} Oct 30 04:43:49.229777 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.229672 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" event={"ID":"76fbbb93-c186-42f6-ba44-fd5b021e5d82","Type":"ContainerStarted","Data":"623cd4aec0148bdeda24181aed98b172236929baad9b2d54c55a2489bad7d027"} Oct 30 04:43:49.230965 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.230937 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq5jn" event={"ID":"801eb2b7-bb95-4b2f-95d6-44880fdf062c","Type":"ContainerStarted","Data":"d6b3d8fa9c46eb9b2e6ff1ce849145653e85316669998c61da7a4da661f836e0"} Oct 30 04:43:49.232281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.232258 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" event={"ID":"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc","Type":"ContainerStarted","Data":"977305a142df87be2d2b6d5959f5e0b8f6a5916123d56a139da72dd307080697"} Oct 30 04:43:49.233861 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.233844 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:43:49.234299 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.234285 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/1.log" Oct 30 04:43:49.234372 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.234312 5058 generic.go:356] "Generic (PLEG): container finished" podID="80515748-a212-4b00-bec6-9b459babaa08" containerID="75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d" exitCode=255 Oct 30 04:43:49.234423 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.234375 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" event={"ID":"80515748-a212-4b00-bec6-9b459babaa08","Type":"ContainerDied","Data":"75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d"} Oct 30 04:43:49.234423 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.234417 5058 scope.go:122] "RemoveContainer" containerID="8a47ad3fa1f53f51ba5fd7bbf87a038a3b56a0edd3622229bee41e5d20b55f5f" Oct 30 04:43:49.234733 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.234709 5058 scope.go:122] "RemoveContainer" containerID="75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d" Oct 30 04:43:49.234941 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.234919 5058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:49.235855 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.235827 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" event={"ID":"5dae5dcf-7d46-4eba-a2c8-1193789e3c28","Type":"ContainerStarted","Data":"2ca47d251b2daa514d9fe5f2856ca9569f165546e0a90027e474e1c5b21f1598"} Oct 30 04:43:49.802761 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.802654 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:49.908901 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.908854 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:49.909054 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.908964 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:49.909054 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.908993 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:49.909054 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.909037 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:49.909198 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.909186 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.90917162 +0000 UTC m=+4.878666588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:49.909391 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.909366 5058 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:49.909464 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.909437 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.909420595 +0000 UTC m=+4.878915570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:49.909540 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.909522 5058 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:49.909604 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:49.909563 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:51.909550561 +0000 UTC m=+4.879045546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:49.912442 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.912415 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5771f103-055c-43b2-b344-fb3f64cc0998-metrics-certs\") pod \"network-metrics-daemon-2fszz\" (UID: \"5771f103-055c-43b2-b344-fb3f64cc0998\") " pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:49.986368 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:49.986330 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fszz" Oct 30 04:43:50.136835 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.136808 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:50.244821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.244791 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:43:50.245332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.244901 5058 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:50.245963 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.245943 5058 scope.go:122] "RemoveContainer" containerID="75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d" Oct 30 04:43:50.246146 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:50.246125 5058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:50.246526 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.246506 5058 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:50.296439 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:50.296412 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fszz"] Oct 30 04:43:50.298529 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:50.298500 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5771f103_055c_43b2_b344_fb3f64cc0998.slice/crio-ffa9228c48672cfed9f387c056756fd07c1c89debc4b965b4e56611412324b4a WatchSource:0}: Error finding container ffa9228c48672cfed9f387c056756fd07c1c89debc4b965b4e56611412324b4a: Status 404 returned error can't find the container with id ffa9228c48672cfed9f387c056756fd07c1c89debc4b965b4e56611412324b4a Oct 30 04:43:51.253385 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.253338 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-84nvd" event={"ID":"5dae5dcf-7d46-4eba-a2c8-1193789e3c28","Type":"ContainerStarted","Data":"abfda5abccfadf6459d82732961b01efb0f55fea3ad3eeb498f5ae6f467cb220"} Oct 30 04:43:51.255365 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.255330 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq5jn" event={"ID":"801eb2b7-bb95-4b2f-95d6-44880fdf062c","Type":"ContainerStarted","Data":"0f28e23479dd2f95eab1dd89523744242159f5a9e6b4b44fa462c90583b89d67"} Oct 30 04:43:51.255486 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.255378 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq5jn" event={"ID":"801eb2b7-bb95-4b2f-95d6-44880fdf062c","Type":"ContainerStarted","Data":"13099f8f0379cc6db1558e69f55a3a15b760f19ee2fa8607f7c354aa56485157"} Oct 30 04:43:51.255486 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.255425 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hq5jn" Oct 30 04:43:51.257087 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.257067 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l2bhc" event={"ID":"1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc","Type":"ContainerStarted","Data":"70ffc725aa167abebf66ce5eb10e39a01ab2c4345d6a1d123c443a8c7efcb2b1"} Oct 30 04:43:51.258418 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.258399 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fszz" event={"ID":"5771f103-055c-43b2-b344-fb3f64cc0998","Type":"ContainerStarted","Data":"ffa9228c48672cfed9f387c056756fd07c1c89debc4b965b4e56611412324b4a"} Oct 30 04:43:51.930771 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.930742 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:51.930905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.930805 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:51.930905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:51.930830 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:51.931031 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:51.930900 5058 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 04:43:51.931031 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:51.930948 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle podName:48383d31-9302-4010-867c-0162421153e4 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:55.93093505 +0000 UTC m=+8.900430018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle") pod "router-default-74b99df667-5m6dw" (UID: "48383d31-9302-4010-867c-0162421153e4") : configmap references non-existent config key: service-ca.crt Oct 30 04:43:51.931031 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:51.930973 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls podName:8a79724a-9e07-48a6-861f-d38ffe1bd578 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:55.930965784 +0000 UTC m=+8.900460751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qp96q" (UID: "8a79724a-9e07-48a6-861f-d38ffe1bd578") : secret "samples-operator-tls" not found Oct 30 04:43:51.931031 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:51.930979 5058 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 04:43:51.931173 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:51.931045 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls podName:09e584bc-955f-4e38-a161-0258bdb03d24 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:55.931026644 +0000 UTC m=+8.900521625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2bq" (UID: "09e584bc-955f-4e38-a161-0258bdb03d24") : secret "insights-runtime-extractor-tls" not found Oct 30 04:43:52.264302 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.264263 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fszz" event={"ID":"5771f103-055c-43b2-b344-fb3f64cc0998","Type":"ContainerStarted","Data":"b266a5e892c85d6220838e44f6a7da113fb1793bda399b4991fd78165fca449d"} Oct 30 04:43:52.264729 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.264336 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fszz" event={"ID":"5771f103-055c-43b2-b344-fb3f64cc0998","Type":"ContainerStarted","Data":"50929765ed2b57e6febc3a614acdf9cd53cce21dcc2d394627921c3f73ce83fa"} Oct 30 04:43:52.265836 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.265811 5058 generic.go:356] "Generic (PLEG): container finished" podID="e11599cb-3adb-4c57-93e5-6a13d5fd50a1" containerID="db5d6ce4071701bca64fc44717d93486526933dc4be82a7d0cb80b685ba5c3a4" exitCode=0 Oct 30 04:43:52.265942 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.265880 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" event={"ID":"e11599cb-3adb-4c57-93e5-6a13d5fd50a1","Type":"ContainerDied","Data":"db5d6ce4071701bca64fc44717d93486526933dc4be82a7d0cb80b685ba5c3a4"} Oct 30 04:43:52.266208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.266194 5058 scope.go:122] "RemoveContainer" containerID="db5d6ce4071701bca64fc44717d93486526933dc4be82a7d0cb80b685ba5c3a4" Oct 30 04:43:52.267592 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.267574 5058 generic.go:356] "Generic (PLEG): container finished" podID="8d84d597-b1ed-41d3-a4a2-1add913c1125" containerID="591e5fe34fea82975462c61acf9be24977336dfe766991ed4abbe97f80ea7084" exitCode=0 Oct 30 04:43:52.267692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.267667 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerDied","Data":"591e5fe34fea82975462c61acf9be24977336dfe766991ed4abbe97f80ea7084"} Oct 30 04:43:52.267995 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:52.267976 5058 scope.go:122] "RemoveContainer" containerID="591e5fe34fea82975462c61acf9be24977336dfe766991ed4abbe97f80ea7084" Oct 30 04:43:53.272799 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:53.272763 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sfq94" event={"ID":"e11599cb-3adb-4c57-93e5-6a13d5fd50a1","Type":"ContainerStarted","Data":"96e17447e44b9991256d0d4fe57f92a440ad11e8cd5a2c2744d0b35a45de85d0"} Oct 30 04:43:53.274518 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:53.274492 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerStarted","Data":"f3101a29ac427457a64ecf281729fa04546fb1261d46627c891a5721f6150e62"} Oct 30 04:43:54.207116 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:54.207076 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:43:55.480371 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.480333 5058 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:43:55.480779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.480720 5058 scope.go:122] "RemoveContainer" containerID="75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d" Oct 30 04:43:55.480917 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:55.480898 5058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wf2dd_openshift-console-operator(80515748-a212-4b00-bec6-9b459babaa08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" Oct 30 04:43:55.971376 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.971336 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:55.971576 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.971430 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:55.971576 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.971468 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:55.972101 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.972078 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48383d31-9302-4010-867c-0162421153e4-service-ca-bundle\") pod \"router-default-74b99df667-5m6dw\" (UID: \"48383d31-9302-4010-867c-0162421153e4\") " pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:55.973767 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.973743 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09e584bc-955f-4e38-a161-0258bdb03d24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2bq\" (UID: \"09e584bc-955f-4e38-a161-0258bdb03d24\") " pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:55.973880 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.973861 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a79724a-9e07-48a6-861f-d38ffe1bd578-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qp96q\" (UID: \"8a79724a-9e07-48a6-861f-d38ffe1bd578\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:55.974669 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.974649 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:55.976751 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.976736 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" Oct 30 04:43:55.989101 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:55.989081 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zh2bq" Oct 30 04:43:56.300618 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:56.300588 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q"] Oct 30 04:43:56.314008 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:56.313981 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2bq"] Oct 30 04:43:56.314727 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:56.314708 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74b99df667-5m6dw"] Oct 30 04:43:56.316435 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:56.316411 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e584bc_955f_4e38_a161_0258bdb03d24.slice/crio-3a692f322e6c849c5deef8eb042fc7e752112f7dd32eb75c26f676e1d16d0a73 WatchSource:0}: Error finding container 3a692f322e6c849c5deef8eb042fc7e752112f7dd32eb75c26f676e1d16d0a73: Status 404 returned error can't find the container with id 3a692f322e6c849c5deef8eb042fc7e752112f7dd32eb75c26f676e1d16d0a73 Oct 30 04:43:56.316668 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:43:56.316642 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48383d31_9302_4010_867c_0162421153e4.slice/crio-ea432a8b584e9b28e1cd53ba019a88e67096dfaabd884633108dd8149cfe7ad3 WatchSource:0}: Error finding container ea432a8b584e9b28e1cd53ba019a88e67096dfaabd884633108dd8149cfe7ad3: Status 404 returned error can't find the container with id ea432a8b584e9b28e1cd53ba019a88e67096dfaabd884633108dd8149cfe7ad3 Oct 30 04:43:56.446095 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:56.446043 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:43:56.709122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:56.709091 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:43:57.292082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.291847 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" event={"ID":"8a79724a-9e07-48a6-861f-d38ffe1bd578","Type":"ContainerStarted","Data":"a141314d1f402a189cb41cd683b3238cde5d5f966cbdf7ea73e1225691223654"} Oct 30 04:43:57.293854 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.293824 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2bq" event={"ID":"09e584bc-955f-4e38-a161-0258bdb03d24","Type":"ContainerStarted","Data":"071233f4f8d72a0d2c7819196e3d858c84575a01d8a58db52046aa8896d9c468"} Oct 30 04:43:57.294000 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.293882 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2bq" event={"ID":"09e584bc-955f-4e38-a161-0258bdb03d24","Type":"ContainerStarted","Data":"869ad6e7ad24e33a94ce949633626f538fa1ff9ac22a776a67849f1c4a03ff94"} Oct 30 04:43:57.294000 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.293909 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2bq" event={"ID":"09e584bc-955f-4e38-a161-0258bdb03d24","Type":"ContainerStarted","Data":"3a692f322e6c849c5deef8eb042fc7e752112f7dd32eb75c26f676e1d16d0a73"} Oct 30 04:43:57.295328 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.295299 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74b99df667-5m6dw" event={"ID":"48383d31-9302-4010-867c-0162421153e4","Type":"ContainerStarted","Data":"f809273be492219c4c1af722ee7be08ececdabf934e3fe544e1cf10938c37202"} Oct 30 04:43:57.295429 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.295344 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74b99df667-5m6dw" event={"ID":"48383d31-9302-4010-867c-0162421153e4","Type":"ContainerStarted","Data":"ea432a8b584e9b28e1cd53ba019a88e67096dfaabd884633108dd8149cfe7ad3"} Oct 30 04:43:57.975567 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.975531 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:57.978594 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:57.978569 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:43:58.020449 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.020419 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:43:58.020668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.020655 5058 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:58.302392 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.302302 5058 generic.go:356] "Generic (PLEG): container finished" podID="8d84d597-b1ed-41d3-a4a2-1add913c1125" containerID="f3101a29ac427457a64ecf281729fa04546fb1261d46627c891a5721f6150e62" exitCode=0 Oct 30 04:43:58.302392 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.302374 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerDied","Data":"f3101a29ac427457a64ecf281729fa04546fb1261d46627c891a5721f6150e62"} Oct 30 04:43:58.302594 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.302461 5058 scope.go:122] "RemoveContainer" containerID="591e5fe34fea82975462c61acf9be24977336dfe766991ed4abbe97f80ea7084" Oct 30 04:43:58.302842 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.302821 5058 scope.go:122] "RemoveContainer" containerID="f3101a29ac427457a64ecf281729fa04546fb1261d46627c891a5721f6150e62" Oct 30 04:43:58.303112 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:43:58.303064 5058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-d82h8_openshift-insights(8d84d597-b1ed-41d3-a4a2-1add913c1125)\"" pod="openshift-insights/insights-operator-7c4986d449-d82h8" podUID="8d84d597-b1ed-41d3-a4a2-1add913c1125" Oct 30 04:43:58.304895 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.304868 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" event={"ID":"8a79724a-9e07-48a6-861f-d38ffe1bd578","Type":"ContainerStarted","Data":"54c05bde65370e7dbc9a5597cdedf4be1c625e2d8aa6dc6f868545c1bd9ebf7e"} Oct 30 04:43:58.305006 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:58.304922 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qp96q" event={"ID":"8a79724a-9e07-48a6-861f-d38ffe1bd578","Type":"ContainerStarted","Data":"89dc1a0bc32c1b6554d7a524873c5dca6bc0e8bcf1825d57b59dbd4bdc7c13e9"} Oct 30 04:43:59.310865 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:59.310826 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2bq" event={"ID":"09e584bc-955f-4e38-a161-0258bdb03d24","Type":"ContainerStarted","Data":"626d136be078ed219aee27b380a421296ab2c0ebf8adc53deeba738fe060ca36"} Oct 30 04:43:59.312410 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:43:59.312391 5058 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:44:00.153557 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.153531 5058 scope.go:122] "RemoveContainer" containerID="75a65ed6495eb8fade45c501af16d8c83d7329a68a3526434fbee0d9ff743d7d" Oct 30 04:44:00.318103 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.318075 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:44:00.318600 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.318188 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" event={"ID":"80515748-a212-4b00-bec6-9b459babaa08","Type":"ContainerStarted","Data":"a55ed1673938f7bec165f477e290c5901812002a7ca015d7d7682cf077f6bad6"} Oct 30 04:44:00.318644 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.318624 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:44:00.319775 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.319750 5058 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-wf2dd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Oct 30 04:44:00.319876 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.319790 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" podUID="80515748-a212-4b00-bec6-9b459babaa08" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Oct 30 04:44:00.456861 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:00.456786 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nrrc2" Oct 30 04:44:01.270386 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:01.270348 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hq5jn" Oct 30 04:44:01.326603 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:01.326574 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wf2dd" Oct 30 04:44:02.160693 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.160652 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:02.164578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.164551 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.190068 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.190036 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:02.196742 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.196711 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh"] Oct 30 04:44:02.200503 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.200480 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:02.203064 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.203040 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fpjq6"] Oct 30 04:44:02.206656 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.206635 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.209967 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.209944 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:44:02.214669 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.214643 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh"] Oct 30 04:44:02.215920 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.215899 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:44:02.218320 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.217541 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74b99df667-5m6dw" Oct 30 04:44:02.228002 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.227977 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38619267-fa1a-42a4-b54c-abf89f69ba9f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9pkhh\" (UID: \"38619267-fa1a-42a4-b54c-abf89f69ba9f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:02.228107 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228021 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53f7eed5-61a2-417e-a55d-90dcb994ad2d-tmp\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.228573 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228554 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.228630 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228582 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.228674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228639 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfp6\" (UniqueName: \"kubernetes.io/projected/53f7eed5-61a2-417e-a55d-90dcb994ad2d-kube-api-access-kbfp6\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.228674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228670 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.228734 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.228686 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hb4\" (UniqueName: \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.231782 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.231483 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fpjq6"] Oct 30 04:44:02.231782 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.231622 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 04:44:02.231782 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.231643 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ppgt5\"" Oct 30 04:44:02.231782 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.231739 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 04:44:02.234039 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.234016 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-znmcp\"" Oct 30 04:44:02.237974 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.237954 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 04:44:02.285580 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.285547 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-646f79545f-ggjr6"] Oct 30 04:44:02.288805 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.288786 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.320489 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.320461 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646f79545f-ggjr6"] Oct 30 04:44:02.329433 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329408 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfp6\" (UniqueName: \"kubernetes.io/projected/53f7eed5-61a2-417e-a55d-90dcb994ad2d-kube-api-access-kbfp6\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329473 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-installation-pull-secrets\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329500 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-bound-sa-token\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329663 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329703 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hb4\" (UniqueName: \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329729 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38619267-fa1a-42a4-b54c-abf89f69ba9f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9pkhh\" (UID: \"38619267-fa1a-42a4-b54c-abf89f69ba9f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329758 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-ca-trust-extracted\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329794 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53f7eed5-61a2-417e-a55d-90dcb994ad2d-tmp\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.329825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.329824 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-tls\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.330230 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330042 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgs4\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-kube-api-access-nhgs4\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.330230 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330098 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.330230 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330166 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.330402 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330210 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53f7eed5-61a2-417e-a55d-90dcb994ad2d-tmp\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.330402 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330260 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-image-registry-private-configuration\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.330402 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330301 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-certificates\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.330402 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330359 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-trusted-ca\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.330579 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330542 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.330579 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.330544 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.331361 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.331340 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.332500 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.332479 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38619267-fa1a-42a4-b54c-abf89f69ba9f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9pkhh\" (UID: \"38619267-fa1a-42a4-b54c-abf89f69ba9f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:02.356350 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.356329 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hb4\" (UniqueName: \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\") pod \"periodic-gathering-95sbp-g28b9\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.357719 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.357698 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfp6\" (UniqueName: \"kubernetes.io/projected/53f7eed5-61a2-417e-a55d-90dcb994ad2d-kube-api-access-kbfp6\") pod \"downloads-6dd789c8b7-fpjq6\" (UID: \"53f7eed5-61a2-417e-a55d-90dcb994ad2d\") " pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.430871 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430791 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-tls\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.430871 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430843 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgs4\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-kube-api-access-nhgs4\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430889 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-image-registry-private-configuration\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430917 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-certificates\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430951 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-trusted-ca\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.430998 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-installation-pull-secrets\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.431025 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-bound-sa-token\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.431068 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-ca-trust-extracted\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.431519 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.431491 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-ca-trust-extracted\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.432081 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.432048 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-certificates\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.432669 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.432648 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-trusted-ca\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.434672 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.434643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-image-registry-private-configuration\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.434755 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.434715 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-installation-pull-secrets\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.435053 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.435035 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-registry-tls\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.452366 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.452337 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-bound-sa-token\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.452577 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.452554 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgs4\" (UniqueName: \"kubernetes.io/projected/0acfde38-04ec-4b8f-b0a9-4a58af82f32d-kube-api-access-nhgs4\") pod \"image-registry-646f79545f-ggjr6\" (UID: \"0acfde38-04ec-4b8f-b0a9-4a58af82f32d\") " pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.476946 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.476914 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:02.518824 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.518791 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:02.527213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.527156 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:02.599883 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.599852 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:02.771085 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.771051 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:02.771825 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:44:02.771799 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd000e60_ef84_46ef_8b6e_cdd55746b44d.slice/crio-0201eff6779fd9c97852869a6cd5b4265b82bb5007ecbab55ab0fb7fcab9ee1f WatchSource:0}: Error finding container 0201eff6779fd9c97852869a6cd5b4265b82bb5007ecbab55ab0fb7fcab9ee1f: Status 404 returned error can't find the container with id 0201eff6779fd9c97852869a6cd5b4265b82bb5007ecbab55ab0fb7fcab9ee1f Oct 30 04:44:02.816957 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.816903 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh"] Oct 30 04:44:02.818010 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:44:02.817986 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38619267_fa1a_42a4_b54c_abf89f69ba9f.slice/crio-08be9ec70a54a39760f0cfd3a90db4f6764ca92e57242b5f359b7180867df883 WatchSource:0}: Error finding container 08be9ec70a54a39760f0cfd3a90db4f6764ca92e57242b5f359b7180867df883: Status 404 returned error can't find the container with id 08be9ec70a54a39760f0cfd3a90db4f6764ca92e57242b5f359b7180867df883 Oct 30 04:44:02.843032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.842035 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fpjq6"] Oct 30 04:44:02.897032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:02.897004 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646f79545f-ggjr6"] Oct 30 04:44:02.898130 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:44:02.898096 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acfde38_04ec_4b8f_b0a9_4a58af82f32d.slice/crio-87cf0191f4c9ba469d422e4bb29b407b321f2b3a3bdb996d05da130a8335c88b WatchSource:0}: Error finding container 87cf0191f4c9ba469d422e4bb29b407b321f2b3a3bdb996d05da130a8335c88b: Status 404 returned error can't find the container with id 87cf0191f4c9ba469d422e4bb29b407b321f2b3a3bdb996d05da130a8335c88b Oct 30 04:44:03.332650 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.332609 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fpjq6" event={"ID":"53f7eed5-61a2-417e-a55d-90dcb994ad2d","Type":"ContainerStarted","Data":"a4adde92a54de51ce999fe1e390011b8e3f9538adf967d01cba82c0f9b182c6d"} Oct 30 04:44:03.334497 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.334425 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95sbp-g28b9" event={"ID":"dd000e60-ef84-46ef-8b6e-cdd55746b44d","Type":"ContainerStarted","Data":"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834"} Oct 30 04:44:03.334497 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.334470 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95sbp-g28b9" event={"ID":"dd000e60-ef84-46ef-8b6e-cdd55746b44d","Type":"ContainerStarted","Data":"0201eff6779fd9c97852869a6cd5b4265b82bb5007ecbab55ab0fb7fcab9ee1f"} Oct 30 04:44:03.336710 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.336660 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" event={"ID":"38619267-fa1a-42a4-b54c-abf89f69ba9f","Type":"ContainerStarted","Data":"08be9ec70a54a39760f0cfd3a90db4f6764ca92e57242b5f359b7180867df883"} Oct 30 04:44:03.339049 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.338620 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" event={"ID":"0acfde38-04ec-4b8f-b0a9-4a58af82f32d","Type":"ContainerStarted","Data":"56592fe9d5a930adfe5bb6e075474783dfee6b708fbf9f44251cf89bddda37e7"} Oct 30 04:44:03.339049 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.338664 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" event={"ID":"0acfde38-04ec-4b8f-b0a9-4a58af82f32d","Type":"ContainerStarted","Data":"87cf0191f4c9ba469d422e4bb29b407b321f2b3a3bdb996d05da130a8335c88b"} Oct 30 04:44:03.339049 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.338866 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:03.374588 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.374524 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-95sbp-g28b9" podStartSLOduration=1.374507245 podStartE2EDuration="1.374507245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:03.374190661 +0000 UTC m=+16.343685649" watchObservedRunningTime="2026-10-30 04:44:03.374507245 +0000 UTC m=+16.344002235" Oct 30 04:44:03.416821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.416765 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" podStartSLOduration=1.416742782 podStartE2EDuration="1.416742782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:03.415858337 +0000 UTC m=+16.385353330" watchObservedRunningTime="2026-10-30 04:44:03.416742782 +0000 UTC m=+16.386237773" Oct 30 04:44:03.839130 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:03.839099 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq5jn_801eb2b7-bb95-4b2f-95d6-44880fdf062c/dns/0.log" Oct 30 04:44:04.037849 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.037806 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq5jn_801eb2b7-bb95-4b2f-95d6-44880fdf062c/kube-rbac-proxy/0.log" Oct 30 04:44:04.212707 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.212664 5058 patch_prober.go:29] interesting pod/image-registry-5dfc9b8cc-wb6qg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 04:44:04.212873 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.212723 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:44:04.345831 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.345793 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" event={"ID":"38619267-fa1a-42a4-b54c-abf89f69ba9f","Type":"ContainerStarted","Data":"8b7091dd7a15d33be25faa48c6eeb725471c80aa1f05a6a27567b3a9b476cf79"} Oct 30 04:44:04.346352 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.346330 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:04.351639 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.351617 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" Oct 30 04:44:04.385654 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.385596 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" podStartSLOduration=1.261124498 podStartE2EDuration="2.385583174s" totalImagesPullingTime="1.124458676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:04.385156462 +0000 UTC m=+17.354651452" watchObservedRunningTime="2026-10-30 04:44:04.385583174 +0000 UTC m=+17.355078163" Oct 30 04:44:04.437537 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.437499 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9k5j_37948e1b-b416-4df5-9c56-7ff2e7996e48/dns-node-resolver/0.log" Oct 30 04:44:04.837284 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:04.837192 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dfc9b8cc-wb6qg_76fbbb93-c186-42f6-ba44-fd5b021e5d82/registry/0.log" Oct 30 04:44:05.038653 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:05.038621 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-646f79545f-ggjr6_0acfde38-04ec-4b8f-b0a9-4a58af82f32d/registry/0.log" Oct 30 04:44:05.237489 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:05.237462 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f6f8cd844-zgvtd_ccd92781-ec40-4093-b4f3-5b91519a86b4/registry/0.log" Oct 30 04:44:05.637566 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:05.637536 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lphjh_bbaf9947-5532-47ce-aea5-dcb456c37684/node-ca/0.log" Oct 30 04:44:06.037306 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:06.037239 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74b99df667-5m6dw_48383d31-9302-4010-867c-0162421153e4/router/0.log" Oct 30 04:44:06.238137 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:06.238103 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-79pwp_a93c592f-d506-49e4-ba78-bdaa2de3561a/serve-healthcheck-canary/0.log" Oct 30 04:44:07.062189 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:07.062151 5058 patch_prober.go:29] interesting pod/image-registry-5dfc9b8cc-wb6qg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 04:44:07.062702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:07.062208 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:44:08.154081 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:08.154046 5058 scope.go:122] "RemoveContainer" containerID="f3101a29ac427457a64ecf281729fa04546fb1261d46627c891a5721f6150e62" Oct 30 04:44:08.369115 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:08.369077 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-d82h8" event={"ID":"8d84d597-b1ed-41d3-a4a2-1add913c1125","Type":"ContainerStarted","Data":"6e218a0a02255942a4c13f2f126b2b5b60f6ebeb4c2a660f2067e541ebf0ff46"} Oct 30 04:44:09.709253 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.709202 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nrmsh"] Oct 30 04:44:09.716279 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.716000 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.746734 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.746701 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 04:44:09.746734 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.746701 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 04:44:09.746918 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.746763 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzdcr\"" Oct 30 04:44:09.796576 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796543 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796762 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796596 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796762 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796736 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-wtmp\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796862 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796782 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-metrics-client-ca\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796862 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796809 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxxz\" (UniqueName: \"kubernetes.io/projected/fa8bdfcb-fb42-4367-abff-dfe545a0628b-kube-api-access-ppxxz\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796862 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796845 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-textfile\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796997 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796874 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-sys\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796997 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796915 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-tls\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.796997 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.796950 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-root\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.808206 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.808176 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 04:44:09.808206 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.808178 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 04:44:09.897812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897776 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-wtmp\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897829 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-metrics-client-ca\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897851 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxxz\" (UniqueName: \"kubernetes.io/projected/fa8bdfcb-fb42-4367-abff-dfe545a0628b-kube-api-access-ppxxz\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897871 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-textfile\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897891 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-sys\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897924 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-tls\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.897975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897944 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-root\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.898191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.897983 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.898191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.898000 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-wtmp\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.898191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.898019 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.898191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.898167 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-sys\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.898471 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.898446 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8bdfcb-fb42-4367-abff-dfe545a0628b-root\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.899336 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.898830 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-metrics-client-ca\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.899336 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.899296 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.899466 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.899394 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-textfile\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.900666 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.900642 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.900983 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.900963 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8bdfcb-fb42-4367-abff-dfe545a0628b-node-exporter-tls\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:09.918731 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:09.918704 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxxz\" (UniqueName: \"kubernetes.io/projected/fa8bdfcb-fb42-4367-abff-dfe545a0628b-kube-api-access-ppxxz\") pod \"node-exporter-nrmsh\" (UID: \"fa8bdfcb-fb42-4367-abff-dfe545a0628b\") " pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:10.045251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:10.045151 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrmsh" Oct 30 04:44:10.101823 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:44:10.101785 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8bdfcb_fb42_4367_abff_dfe545a0628b.slice/crio-3bc0065b2370b3d14249e13b01a3b9a51b9ea4f853aea214ce2026ce00a37849 WatchSource:0}: Error finding container 3bc0065b2370b3d14249e13b01a3b9a51b9ea4f853aea214ce2026ce00a37849: Status 404 returned error can't find the container with id 3bc0065b2370b3d14249e13b01a3b9a51b9ea4f853aea214ce2026ce00a37849 Oct 30 04:44:10.381691 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:10.381607 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrmsh" event={"ID":"fa8bdfcb-fb42-4367-abff-dfe545a0628b","Type":"ContainerStarted","Data":"3bc0065b2370b3d14249e13b01a3b9a51b9ea4f853aea214ce2026ce00a37849"} Oct 30 04:44:12.217020 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:12.216989 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:44:12.393332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:12.393294 5058 generic.go:356] "Generic (PLEG): container finished" podID="fa8bdfcb-fb42-4367-abff-dfe545a0628b" containerID="2b43d9af357f25403786e19b7c765a40eb9fe150f43ef64abcbf3821e1b7765e" exitCode=0 Oct 30 04:44:12.393506 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:12.393360 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrmsh" event={"ID":"fa8bdfcb-fb42-4367-abff-dfe545a0628b","Type":"ContainerDied","Data":"2b43d9af357f25403786e19b7c765a40eb9fe150f43ef64abcbf3821e1b7765e"} Oct 30 04:44:13.400791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:13.400752 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrmsh" event={"ID":"fa8bdfcb-fb42-4367-abff-dfe545a0628b","Type":"ContainerStarted","Data":"6e84dcbde1ec03ddc814cee1d50a74a0bb40248308314816dd7bce16572eabbe"} Oct 30 04:44:13.401313 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:13.400822 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrmsh" event={"ID":"fa8bdfcb-fb42-4367-abff-dfe545a0628b","Type":"ContainerStarted","Data":"ebcf8d6442f678bc3cb6ac9bec6ae4b9aee5cc7fc58867f1abce7549900739f0"} Oct 30 04:44:13.457645 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:13.457594 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nrmsh" podStartSLOduration=2.994231051 podStartE2EDuration="4.457572774s" totalImagesPullingTime="1.382874441s" totalInitContainerRuntime="80.467282ms" isStatefulPod=true podCreationTimestamp="2026-10-30 04:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:13.455743093 +0000 UTC m=+26.425238105" watchObservedRunningTime="2026-10-30 04:44:13.457572774 +0000 UTC m=+26.427067809" Oct 30 04:44:14.212831 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:14.212803 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:44:15.819789 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:15.819761 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s445b" Oct 30 04:44:16.448917 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:16.448885 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6v95n" Oct 30 04:44:19.190198 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:44:19.190156 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:44:19.430603 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:19.430555 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fpjq6" event={"ID":"53f7eed5-61a2-417e-a55d-90dcb994ad2d","Type":"ContainerStarted","Data":"0189e8c2116b06a345c5a3b4c62f37ec1e79318d921a97ed09952b303e4ab58c"} Oct 30 04:44:19.431454 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:19.431378 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:19.449568 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:19.449536 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fpjq6" Oct 30 04:44:19.485941 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:19.485881 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fpjq6" podStartSLOduration=1.378724088 podStartE2EDuration="17.485863162s" totalImagesPullingTime="16.107139074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:19.484490186 +0000 UTC m=+32.453985177" watchObservedRunningTime="2026-10-30 04:44:19.485863162 +0000 UTC m=+32.455358152" Oct 30 04:44:24.362283 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:24.359885 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-646f79545f-ggjr6" Oct 30 04:44:24.449870 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:24.449823 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:44:27.235368 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.235323 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" podUID="ccd92781-ec40-4093-b4f3-5b91519a86b4" containerName="registry" containerID="cri-o://9e69c2af3cc937f1325decfa8dc43be2a9fa75267baf8e5041dff89e5241ddaa" gracePeriod=30 Oct 30 04:44:27.490275 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.490240 5058 generic.go:356] "Generic (PLEG): container finished" podID="ccd92781-ec40-4093-b4f3-5b91519a86b4" containerID="9e69c2af3cc937f1325decfa8dc43be2a9fa75267baf8e5041dff89e5241ddaa" exitCode=0 Oct 30 04:44:27.490480 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.490395 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" event={"ID":"ccd92781-ec40-4093-b4f3-5b91519a86b4","Type":"ContainerDied","Data":"9e69c2af3cc937f1325decfa8dc43be2a9fa75267baf8e5041dff89e5241ddaa"} Oct 30 04:44:27.517201 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.517172 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:44:27.597962 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.597924 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598116 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.597990 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598116 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598029 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598285 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598260 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598363 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598325 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598358 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598396 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598521 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598467 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") pod \"ccd92781-ec40-4093-b4f3-5b91519a86b4\" (UID: \"ccd92781-ec40-4093-b4f3-5b91519a86b4\") " Oct 30 04:44:27.598749 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598625 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:27.598858 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.598825 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:27.604456 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.601303 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:27.604456 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.601664 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:27.604456 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.601723 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:27.604456 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.602686 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "kube-api-access-5wzsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:27.604456 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.602915 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:27.607825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.607792 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted" pod "ccd92781-ec40-4093-b4f3-5b91519a86b4" (UID: "ccd92781-ec40-4093-b4f3-5b91519a86b4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:27.699291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699257 5058 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-certificates\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699287 5058 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-installation-pull-secrets\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699298 5058 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccd92781-ec40-4093-b4f3-5b91519a86b4-image-registry-private-configuration\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699528 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699309 5058 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccd92781-ec40-4093-b4f3-5b91519a86b4-trusted-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699528 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699318 5058 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-bound-sa-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699528 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699326 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzsf\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-kube-api-access-5wzsf\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699528 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699334 5058 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccd92781-ec40-4093-b4f3-5b91519a86b4-ca-trust-extracted\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:27.699528 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:27.699343 5058 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccd92781-ec40-4093-b4f3-5b91519a86b4-registry-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:28.496129 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:28.496098 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" Oct 30 04:44:28.496129 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:28.496107 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6f8cd844-zgvtd" event={"ID":"ccd92781-ec40-4093-b4f3-5b91519a86b4","Type":"ContainerDied","Data":"e5937e6b9c4bf0c76bbe1f2e2f7dfd84bd0ea7a64e240ebfd81832bc85af78df"} Oct 30 04:44:28.496624 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:28.496169 5058 scope.go:122] "RemoveContainer" containerID="9e69c2af3cc937f1325decfa8dc43be2a9fa75267baf8e5041dff89e5241ddaa" Oct 30 04:44:28.562351 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:28.562315 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:44:28.575622 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:28.575596 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f6f8cd844-zgvtd"] Oct 30 04:44:29.158641 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:29.158604 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd92781-ec40-4093-b4f3-5b91519a86b4" path="/var/lib/kubelet/pods/ccd92781-ec40-4093-b4f3-5b91519a86b4/volumes" Oct 30 04:44:35.008918 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.008870 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:35.009482 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.009111 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-95sbp-g28b9" podUID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" containerName="insights-gathering" containerID="cri-o://06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834" gracePeriod=30 Oct 30 04:44:35.009905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.009884 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:44:35.010089 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.010056 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" podUID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" containerName="insights-gathering" containerID="cri-o://ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6" gracePeriod=30 Oct 30 04:44:35.317313 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.317281 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:35.335342 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.335311 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:44:35.471804 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471767 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\") pod \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " Oct 30 04:44:35.471981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471823 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") pod \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " Oct 30 04:44:35.471981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471850 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\" (UniqueName: \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\") pod \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " Oct 30 04:44:35.471981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471871 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\") pod \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " Oct 30 04:44:35.471981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471918 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\") pod \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\" (UID: \"dd000e60-ef84-46ef-8b6e-cdd55746b44d\") " Oct 30 04:44:35.471981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471961 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") pod \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " Oct 30 04:44:35.472348 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.471994 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") pod \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " Oct 30 04:44:35.472348 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472024 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") pod \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\" (UID: \"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88\") " Oct 30 04:44:35.472348 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472262 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path" pod "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" (UID: "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:35.472348 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472169 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path" pod "dd000e60-ef84-46ef-8b6e-cdd55746b44d" (UID: "dd000e60-ef84-46ef-8b6e-cdd55746b44d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:35.472558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472437 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle" pod "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" (UID: "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:35.472558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472454 5058 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dd000e60-ef84-46ef-8b6e-cdd55746b44d-archives-path\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.472558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472476 5058 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-archives-path\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.472558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472520 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle" pod "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" (UID: "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:35.472558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472518 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle" pod "dd000e60-ef84-46ef-8b6e-cdd55746b44d" (UID: "dd000e60-ef84-46ef-8b6e-cdd55746b44d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:35.472719 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.472562 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle" pod "dd000e60-ef84-46ef-8b6e-cdd55746b44d" (UID: "dd000e60-ef84-46ef-8b6e-cdd55746b44d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:35.474454 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.474427 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4" pod "dd000e60-ef84-46ef-8b6e-cdd55746b44d" (UID: "dd000e60-ef84-46ef-8b6e-cdd55746b44d"). InnerVolumeSpecName "kube-api-access-s9hb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:35.474564 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.474461 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km" pod "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" (UID: "35c4081d-a7a9-4c3b-8f6d-bd50698a6b88"). InnerVolumeSpecName "kube-api-access-b99km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:35.529651 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.529553 5058 generic.go:356] "Generic (PLEG): container finished" podID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" containerID="06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834" exitCode=2 Oct 30 04:44:35.529651 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.529625 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95sbp-g28b9" Oct 30 04:44:35.529943 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.529642 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95sbp-g28b9" event={"ID":"dd000e60-ef84-46ef-8b6e-cdd55746b44d","Type":"ContainerDied","Data":"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834"} Oct 30 04:44:35.529943 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.529727 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95sbp-g28b9" event={"ID":"dd000e60-ef84-46ef-8b6e-cdd55746b44d","Type":"ContainerDied","Data":"0201eff6779fd9c97852869a6cd5b4265b82bb5007ecbab55ab0fb7fcab9ee1f"} Oct 30 04:44:35.529943 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.529758 5058 scope.go:122] "RemoveContainer" containerID="06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834" Oct 30 04:44:35.531044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.531009 5058 generic.go:356] "Generic (PLEG): container finished" podID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" containerID="ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6" exitCode=2 Oct 30 04:44:35.531178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.531058 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" event={"ID":"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88","Type":"ContainerDied","Data":"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6"} Oct 30 04:44:35.531178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.531108 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" event={"ID":"35c4081d-a7a9-4c3b-8f6d-bd50698a6b88","Type":"ContainerDied","Data":"9cc93acf470b3d57ccdcd1c2b7bd47ac6c852d4aee3c5495750bc7183a16008b"} Oct 30 04:44:35.531178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.531067 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kz2hv-m8wbg" Oct 30 04:44:35.539513 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.539496 5058 scope.go:122] "RemoveContainer" containerID="06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834" Oct 30 04:44:35.539804 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:44:35.539784 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834\": container with ID starting with 06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834 not found: ID does not exist" containerID="06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834" Oct 30 04:44:35.539888 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.539812 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834"} err="failed to get container status \"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834\": rpc error: code = NotFound desc = could not find container \"06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834\": container with ID starting with 06c1f29b4ddd0d0f927e82ad0dc56d6379894af4e20d6467dc312a26feb0f834 not found: ID does not exist" Oct 30 04:44:35.539888 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.539842 5058 scope.go:122] "RemoveContainer" containerID="ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6" Oct 30 04:44:35.547415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.547398 5058 scope.go:122] "RemoveContainer" containerID="ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6" Oct 30 04:44:35.547684 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:44:35.547657 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6\": container with ID starting with ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6 not found: ID does not exist" containerID="ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6" Oct 30 04:44:35.547775 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.547702 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6"} err="failed to get container status \"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6\": rpc error: code = NotFound desc = could not find container \"ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6\": container with ID starting with ce15f8cc5244d27f831116ab74260e9d884fcc22d8df698dd29dbd89af4f06c6 not found: ID does not exist" Oct 30 04:44:35.573576 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573541 5058 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-trusted-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.573726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573581 5058 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-service-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.573726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573595 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99km\" (UniqueName: \"kubernetes.io/projected/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88-kube-api-access-b99km\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.573726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573604 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9hb4\" (UniqueName: \"kubernetes.io/projected/dd000e60-ef84-46ef-8b6e-cdd55746b44d-kube-api-access-s9hb4\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.573726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573613 5058 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-service-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.573726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.573621 5058 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd000e60-ef84-46ef-8b6e-cdd55746b44d-trusted-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:35.588127 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.588099 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:35.591101 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.591074 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-95sbp-g28b9"] Oct 30 04:44:35.625608 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.625565 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:44:35.629275 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:35.629245 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-kz2hv-m8wbg"] Oct 30 04:44:37.158981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:37.158945 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" path="/var/lib/kubelet/pods/35c4081d-a7a9-4c3b-8f6d-bd50698a6b88/volumes" Oct 30 04:44:37.159365 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:37.159355 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" path="/var/lib/kubelet/pods/dd000e60-ef84-46ef-8b6e-cdd55746b44d/volumes" Oct 30 04:44:49.189498 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:44:49.189459 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:44:49.479559 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.479447 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" containerID="cri-o://4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c" gracePeriod=30 Oct 30 04:44:49.759746 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.759677 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:44:49.791851 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791805 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791863 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791914 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791954 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791980 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.791998 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.792030 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.792065 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") pod \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\" (UID: \"76fbbb93-c186-42f6-ba44-fd5b021e5d82\") " Oct 30 04:44:49.792758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.792529 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:49.792758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.792692 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:49.794660 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.794619 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:49.794766 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.794665 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:49.794766 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.794734 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "kube-api-access-79bn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:49.794898 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.794877 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:49.795414 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.795391 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:49.799189 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.799165 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted" pod "76fbbb93-c186-42f6-ba44-fd5b021e5d82" (UID: "76fbbb93-c186-42f6-ba44-fd5b021e5d82"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:49.893298 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893259 5058 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-trusted-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893298 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893295 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bn8\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-kube-api-access-79bn8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893298 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893307 5058 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-certificates\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893316 5058 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-registry-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893326 5058 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76fbbb93-c186-42f6-ba44-fd5b021e5d82-bound-sa-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893334 5058 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76fbbb93-c186-42f6-ba44-fd5b021e5d82-ca-trust-extracted\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893344 5058 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-image-registry-private-configuration\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:49.893531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:49.893354 5058 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76fbbb93-c186-42f6-ba44-fd5b021e5d82-installation-pull-secrets\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:44:50.597795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.597758 5058 generic.go:356] "Generic (PLEG): container finished" podID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerID="4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c" exitCode=0 Oct 30 04:44:50.598208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.597821 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" event={"ID":"76fbbb93-c186-42f6-ba44-fd5b021e5d82","Type":"ContainerDied","Data":"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c"} Oct 30 04:44:50.598208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.597865 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" event={"ID":"76fbbb93-c186-42f6-ba44-fd5b021e5d82","Type":"ContainerDied","Data":"623cd4aec0148bdeda24181aed98b172236929baad9b2d54c55a2489bad7d027"} Oct 30 04:44:50.598208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.597884 5058 scope.go:122] "RemoveContainer" containerID="4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c" Oct 30 04:44:50.598208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.597828 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfc9b8cc-wb6qg" Oct 30 04:44:50.606743 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.606724 5058 scope.go:122] "RemoveContainer" containerID="4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c" Oct 30 04:44:50.607004 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:44:50.606982 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c\": container with ID starting with 4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c not found: ID does not exist" containerID="4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c" Oct 30 04:44:50.607044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.607014 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c"} err="failed to get container status \"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c\": rpc error: code = NotFound desc = could not find container \"4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c\": container with ID starting with 4871bfe849f834945dbf6a7a07e2e6a0b51d3efe9ec8f14b68209eaf3f274d5c not found: ID does not exist" Oct 30 04:44:50.664032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.664002 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:44:50.677480 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:50.677454 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dfc9b8cc-wb6qg"] Oct 30 04:44:51.159116 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:51.159077 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" path="/var/lib/kubelet/pods/76fbbb93-c186-42f6-ba44-fd5b021e5d82/volumes" Oct 30 04:44:52.607688 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:52.607617 5058 generic.go:356] "Generic (PLEG): container finished" podID="64040d3f-1de7-42d3-a9aa-6fefa2a926ed" containerID="de88270bb0d30814b8c5cf9cd7d331ce7a22408d8d64401f1f4b3d3da592e0e2" exitCode=0 Oct 30 04:44:52.608038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:52.607685 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" event={"ID":"64040d3f-1de7-42d3-a9aa-6fefa2a926ed","Type":"ContainerDied","Data":"de88270bb0d30814b8c5cf9cd7d331ce7a22408d8d64401f1f4b3d3da592e0e2"} Oct 30 04:44:52.608038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:52.607999 5058 scope.go:122] "RemoveContainer" containerID="de88270bb0d30814b8c5cf9cd7d331ce7a22408d8d64401f1f4b3d3da592e0e2" Oct 30 04:44:53.613048 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:44:53.613013 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-krmmv" event={"ID":"64040d3f-1de7-42d3-a9aa-6fefa2a926ed","Type":"ContainerStarted","Data":"517de1595b7ddb3399eb833744d1a7b1b407beb2222d8d76a4883e99d92613c4"} Oct 30 04:45:19.188411 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:45:19.188372 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:45:49.189325 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:45:49.189282 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:46:19.189361 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:46:19.189280 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:46:49.189528 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:46:49.189488 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:47:19.190412 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:47:19.190368 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:47:49.188688 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:47:49.188647 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:48:19.188689 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:48:19.188648 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:48:47.194694 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:47.194663 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:48:47.197824 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:47.197797 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:48:49.189445 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:48:49.189406 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:48:53.083227 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083181 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn"] Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083517 5058 cpu_manager.go:414] "containerMap: removing container" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083529 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083544 5058 cpu_manager.go:414] "containerMap: removing container" podUID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083549 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083556 5058 cpu_manager.go:414] "containerMap: removing container" podUID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083562 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083574 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ccd92781-ec40-4093-b4f3-5b91519a86b4" containerName="registry" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083579 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd92781-ec40-4093-b4f3-5b91519a86b4" containerName="registry" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083642 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fbbb93-c186-42f6-ba44-fd5b021e5d82" containerName="registry" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083651 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd000e60-ef84-46ef-8b6e-cdd55746b44d" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083656 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c4081d-a7a9-4c3b-8f6d-bd50698a6b88" containerName="insights-gathering" Oct 30 04:48:53.083697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.083662 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd92781-ec40-4093-b4f3-5b91519a86b4" containerName="registry" Oct 30 04:48:53.086758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.086740 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.122281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.122248 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 04:48:53.122458 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.122255 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 04:48:53.123714 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.123689 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn"] Oct 30 04:48:53.155423 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155385 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.155607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155435 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbpb\" (UniqueName: \"kubernetes.io/projected/faac0405-0672-447b-be1c-661e91ace85a-kube-api-access-6vbpb\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.155607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155576 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.155736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155609 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.155736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155668 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.155849 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.155740 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.208479 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.208444 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 04:48:53.208658 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.208583 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wr5jk\"" Oct 30 04:48:53.208658 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.208596 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 04:48:53.208658 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.208638 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 04:48:53.256531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256486 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.256723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256540 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.256723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.256723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256605 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbpb\" (UniqueName: \"kubernetes.io/projected/faac0405-0672-447b-be1c-661e91ace85a-kube-api-access-6vbpb\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.256723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256663 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.256723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.256700 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.257159 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.257126 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.259031 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.259008 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faac0405-0672-447b-be1c-661e91ace85a-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.259293 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.259269 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.259293 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.259285 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.259375 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.259302 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/faac0405-0672-447b-be1c-661e91ace85a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.277697 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.277669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbpb\" (UniqueName: \"kubernetes.io/projected/faac0405-0672-447b-be1c-661e91ace85a-kube-api-access-6vbpb\") pod \"opendatahub-operator-controller-manager-8484dd48c9-xqtxn\" (UID: \"faac0405-0672-447b-be1c-661e91ace85a\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.402120 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.402078 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:53.753335 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:48:53.753298 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaac0405_0672_447b_be1c_661e91ace85a.slice/crio-7d707b39216623273a2899c650f40e0ba98d1054aa933195845d9609cc529a3b WatchSource:0}: Error finding container 7d707b39216623273a2899c650f40e0ba98d1054aa933195845d9609cc529a3b: Status 404 returned error can't find the container with id 7d707b39216623273a2899c650f40e0ba98d1054aa933195845d9609cc529a3b Oct 30 04:48:53.754956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.754933 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:48:53.755774 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:53.755750 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn"] Oct 30 04:48:54.554148 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:54.554058 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" event={"ID":"faac0405-0672-447b-be1c-661e91ace85a","Type":"ContainerStarted","Data":"7d707b39216623273a2899c650f40e0ba98d1054aa933195845d9609cc529a3b"} Oct 30 04:48:57.569455 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:57.569412 5058 generic.go:356] "Generic (PLEG): container finished" podID="faac0405-0672-447b-be1c-661e91ace85a" containerID="09887482030a3b86a3076c48a391894e58667b408201a9029c87071a0abdbf6b" exitCode=0 Oct 30 04:48:57.569892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:57.569479 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" event={"ID":"faac0405-0672-447b-be1c-661e91ace85a","Type":"ContainerDied","Data":"09887482030a3b86a3076c48a391894e58667b408201a9029c87071a0abdbf6b"} Oct 30 04:48:58.575373 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:58.575334 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" event={"ID":"faac0405-0672-447b-be1c-661e91ace85a","Type":"ContainerStarted","Data":"535989cc534a6f3c32093be386b9f1a06e2dba4a648302934ea66a5372c18cdc"} Oct 30 04:48:58.575854 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:58.575451 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:48:58.660881 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:48:58.660822 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" podStartSLOduration=2.006757284 podStartE2EDuration="5.660808803s" totalImagesPullingTime="3.111016168s" totalInitContainerRuntime="543.035351ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:58.64632952 +0000 UTC m=+311.615824509" watchObservedRunningTime="2026-10-30 04:48:58.660808803 +0000 UTC m=+311.630303791" Oct 30 04:49:09.582034 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:09.582000 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn" Oct 30 04:49:15.782963 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.782922 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-gm2nc"] Oct 30 04:49:15.792864 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.792833 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:15.805637 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.805596 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-gm2nc"] Oct 30 04:49:15.816976 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.816915 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 04:49:15.817155 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.817052 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-9cj9f\"" Oct 30 04:49:15.832231 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.832185 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckds\" (UniqueName: \"kubernetes.io/projected/cc2dd475-d48d-43e5-a32b-721786ce16b1-kube-api-access-5ckds\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:15.832392 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.832351 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:15.856933 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.856895 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4"] Oct 30 04:49:15.860478 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.860453 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:15.880762 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.880730 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4"] Oct 30 04:49:15.910896 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.910866 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 04:49:15.910896 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.910875 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 04:49:15.911117 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.910960 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hmrrz\"" Oct 30 04:49:15.933030 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.932995 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287a26ad-2736-4bd4-a588-f384ad7de793-tmp\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:15.933030 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.933030 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96b5b\" (UniqueName: \"kubernetes.io/projected/287a26ad-2736-4bd4-a588-f384ad7de793-kube-api-access-96b5b\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:15.933295 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.933142 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckds\" (UniqueName: \"kubernetes.io/projected/cc2dd475-d48d-43e5-a32b-721786ce16b1-kube-api-access-5ckds\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:15.933295 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.933259 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:15.933400 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.933316 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287a26ad-2736-4bd4-a588-f384ad7de793-tls-certs\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:15.933456 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:49:15.933392 5058 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 04:49:15.933456 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:49:15.933453 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs podName:cc2dd475-d48d-43e5-a32b-721786ce16b1 nodeName:}" failed. No retries permitted until 2026-10-30 04:49:16.433436242 +0000 UTC m=+329.402931229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs") pod "odh-observability-584f99c788-gm2nc" (UID: "cc2dd475-d48d-43e5-a32b-721786ce16b1") : secret "odh-observability-webhook-cert" not found Oct 30 04:49:15.955113 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:15.955077 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckds\" (UniqueName: \"kubernetes.io/projected/cc2dd475-d48d-43e5-a32b-721786ce16b1-kube-api-access-5ckds\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:16.034045 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.033947 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287a26ad-2736-4bd4-a588-f384ad7de793-tls-certs\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.034045 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.033998 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287a26ad-2736-4bd4-a588-f384ad7de793-tmp\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.034332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.034115 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96b5b\" (UniqueName: \"kubernetes.io/projected/287a26ad-2736-4bd4-a588-f384ad7de793-kube-api-access-96b5b\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.036241 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.036195 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287a26ad-2736-4bd4-a588-f384ad7de793-tmp\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.036492 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.036471 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287a26ad-2736-4bd4-a588-f384ad7de793-tls-certs\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.054094 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.054062 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96b5b\" (UniqueName: \"kubernetes.io/projected/287a26ad-2736-4bd4-a588-f384ad7de793-kube-api-access-96b5b\") pod \"kube-auth-proxy-54668d58ff-kgtw4\" (UID: \"287a26ad-2736-4bd4-a588-f384ad7de793\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.172473 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.172428 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" Oct 30 04:49:16.438320 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.438276 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:16.440773 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.440749 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/cc2dd475-d48d-43e5-a32b-721786ce16b1-webhook-certs\") pod \"odh-observability-584f99c788-gm2nc\" (UID: \"cc2dd475-d48d-43e5-a32b-721786ce16b1\") " pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:16.463871 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.463840 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4"] Oct 30 04:49:16.464681 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:49:16.464658 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287a26ad_2736_4bd4_a588_f384ad7de793.slice/crio-34fdefa64079cee10588912ce532eafbff67788800040cefc8b508b838fed66f WatchSource:0}: Error finding container 34fdefa64079cee10588912ce532eafbff67788800040cefc8b508b838fed66f: Status 404 returned error can't find the container with id 34fdefa64079cee10588912ce532eafbff67788800040cefc8b508b838fed66f Oct 30 04:49:16.644953 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.644918 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" event={"ID":"287a26ad-2736-4bd4-a588-f384ad7de793","Type":"ContainerStarted","Data":"34fdefa64079cee10588912ce532eafbff67788800040cefc8b508b838fed66f"} Oct 30 04:49:16.706361 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.706265 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:16.987654 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.987570 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:49:16.991119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:16.991100 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.009411 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.009384 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 04:49:17.010846 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.010820 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-gm2nc"] Oct 30 04:49:17.011618 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:49:17.011590 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2dd475_d48d_43e5_a32b_721786ce16b1.slice/crio-e5413f2d5cb90d557510502419a147f88dedf31b616b01fd7fc8bebd42ca27f5 WatchSource:0}: Error finding container e5413f2d5cb90d557510502419a147f88dedf31b616b01fd7fc8bebd42ca27f5: Status 404 returned error can't find the container with id e5413f2d5cb90d557510502419a147f88dedf31b616b01fd7fc8bebd42ca27f5 Oct 30 04:49:17.022908 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.022876 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 04:49:17.023035 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.022893 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 04:49:17.023287 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.023265 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8lhdl\"" Oct 30 04:49:17.023395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.023292 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 04:49:17.024509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.024488 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:49:17.044014 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.043983 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044168 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044032 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b289x\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044168 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044122 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044168 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044161 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044290 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044343 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044329 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.044476 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.044458 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145757 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145712 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b289x\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145769 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145805 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145844 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145886 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.145955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145937 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.146192 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.145974 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.147192 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.147161 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.149782 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.149608 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.150990 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.150970 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.151159 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.151140 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.154464 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.154441 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.168283 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.168207 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.168421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.168284 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b289x\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\") pod \"istiod-openshift-gateway-7876bfcb74-lr8nf\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.303412 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.303315 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:17.624795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.624762 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:49:17.626629 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:49:17.626595 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5bd6dc_c151_418a_9828_a614e5f17063.slice/crio-80f1937bbc9f07f95c1d002126241f03e7f46d66bf6ec6cbce4caabe1264c705 WatchSource:0}: Error finding container 80f1937bbc9f07f95c1d002126241f03e7f46d66bf6ec6cbce4caabe1264c705: Status 404 returned error can't find the container with id 80f1937bbc9f07f95c1d002126241f03e7f46d66bf6ec6cbce4caabe1264c705 Oct 30 04:49:17.651475 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.651433 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-gm2nc" event={"ID":"cc2dd475-d48d-43e5-a32b-721786ce16b1","Type":"ContainerStarted","Data":"e5413f2d5cb90d557510502419a147f88dedf31b616b01fd7fc8bebd42ca27f5"} Oct 30 04:49:17.652692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:17.652662 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" event={"ID":"cd5bd6dc-c151-418a-9828-a614e5f17063","Type":"ContainerStarted","Data":"80f1937bbc9f07f95c1d002126241f03e7f46d66bf6ec6cbce4caabe1264c705"} Oct 30 04:49:19.196559 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:49:19.196491 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:49:19.666627 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:19.666582 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" event={"ID":"287a26ad-2736-4bd4-a588-f384ad7de793","Type":"ContainerStarted","Data":"8e83404026e927d5fdded554cad977f9e9cd37fc2e7ecb7ca9eed8c5b5f4784c"} Oct 30 04:49:19.706826 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:19.706773 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4" podStartSLOduration=1.892853959 podStartE2EDuration="4.706757698s" totalImagesPullingTime="2.813903739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:19.706697025 +0000 UTC m=+332.676192018" watchObservedRunningTime="2026-10-30 04:49:19.706757698 +0000 UTC m=+332.676252684" Oct 30 04:49:20.502156 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.502123 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:49:20.502457 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.502190 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:49:20.672626 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.672584 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-gm2nc" event={"ID":"cc2dd475-d48d-43e5-a32b-721786ce16b1","Type":"ContainerStarted","Data":"82465c5645cd465bcdb88d2e57515fb0f132c14d99b79423bf1e7d2c31ac0a1e"} Oct 30 04:49:20.672794 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.672656 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:20.680059 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.680011 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" event={"ID":"cd5bd6dc-c151-418a-9828-a614e5f17063","Type":"ContainerStarted","Data":"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e"} Oct 30 04:49:20.713097 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.713035 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-gm2nc" podStartSLOduration=2.824536695 podStartE2EDuration="5.713018966s" totalImagesPullingTime="2.888482271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:20.712153666 +0000 UTC m=+333.681648654" watchObservedRunningTime="2026-10-30 04:49:20.713018966 +0000 UTC m=+333.682513954" Oct 30 04:49:20.754675 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:20.754553 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" podStartSLOduration=1.883316502 podStartE2EDuration="4.754532571s" totalImagesPullingTime="2.871216069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:20.754170249 +0000 UTC m=+333.723665238" watchObservedRunningTime="2026-10-30 04:49:20.754532571 +0000 UTC m=+333.724027560" Oct 30 04:49:21.685287 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:21.685251 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:21.687088 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:21.687064 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:49:31.687801 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:49:31.687764 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-gm2nc" Oct 30 04:49:49.189603 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:49:49.189564 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:50:14.191974 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.191937 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 04:50:14.195385 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.195368 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.217315 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.217282 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 04:50:14.239833 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.239739 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlq7\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.239833 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.239821 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.240143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.239921 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.240143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.239969 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.240143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.240087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.240371 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.240192 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.240371 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.240244 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.340816 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.340770 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlq7\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.340816 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.340823 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.341084 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.340849 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.341084 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.341008 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.341084 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.341078 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.341310 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.341126 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.341310 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.341166 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.342296 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.342265 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.343623 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.343595 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.343816 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.343793 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.343885 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.343830 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.344053 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.344033 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.361170 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.361136 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.361365 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.361343 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlq7\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-842vl\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.506706 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.506624 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.811267 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.811234 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 04:50:14.812276 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:50:14.812213 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d5b74b_ff1d_404f_850f_ab1f898a198f.slice/crio-9a81fdc51bbffc75d46922cf9b421d4b018a57047af513bcfacc0bf08f9b899b WatchSource:0}: Error finding container 9a81fdc51bbffc75d46922cf9b421d4b018a57047af513bcfacc0bf08f9b899b: Status 404 returned error can't find the container with id 9a81fdc51bbffc75d46922cf9b421d4b018a57047af513bcfacc0bf08f9b899b Oct 30 04:50:14.814312 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.814274 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:50:14.814377 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.814353 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:50:14.911529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.911488 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" event={"ID":"12d5b74b-ff1d-404f-850f-ab1f898a198f","Type":"ContainerStarted","Data":"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c"} Oct 30 04:50:14.911702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.911546 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" event={"ID":"12d5b74b-ff1d-404f-850f-ab1f898a198f","Type":"ContainerStarted","Data":"9a81fdc51bbffc75d46922cf9b421d4b018a57047af513bcfacc0bf08f9b899b"} Oct 30 04:50:14.911702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.911616 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:14.955193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:14.955113 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" podStartSLOduration=0.955099653 podStartE2EDuration="955.099653ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:14.954701906 +0000 UTC m=+387.924196895" watchObservedRunningTime="2026-10-30 04:50:14.955099653 +0000 UTC m=+387.924594642" Oct 30 04:50:15.917675 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:15.917642 5058 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-842vl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:50:15.918083 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:15.917695 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:50:18.917522 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:18.917489 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 04:50:19.005598 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.005564 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:50:19.005783 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.005762 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" podUID="cd5bd6dc-c151-418a-9828-a614e5f17063" containerName="discovery" containerID="cri-o://39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e" gracePeriod=30 Oct 30 04:50:19.194755 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:50:19.194712 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:50:19.252453 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.252425 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:50:19.287134 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287101 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287183 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287238 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287271 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287317 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287353 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287530 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\") pod \"cd5bd6dc-c151-418a-9828-a614e5f17063\" (UID: \"cd5bd6dc-c151-418a-9828-a614e5f17063\") " Oct 30 04:50:19.287768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287726 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:50:19.287882 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.287861 5058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-ca-configmap\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.289623 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.289597 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:50:19.289711 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.289632 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:50:19.289834 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.289805 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:50:19.289935 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.289844 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "kube-api-access-b289x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:50:19.290044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.290024 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:50:19.290105 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.290088 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs" pod "cd5bd6dc-c151-418a-9828-a614e5f17063" (UID: "cd5bd6dc-c151-418a-9828-a614e5f17063"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:50:19.388395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388352 5058 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-cacerts\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.388395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388387 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b289x\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-kube-api-access-b289x\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.388395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388398 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.388395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388406 5058 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-kubeconfig\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.388665 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388416 5058 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5bd6dc-c151-418a-9828-a614e5f17063-local-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.388665 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.388424 5058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd5bd6dc-c151-418a-9828-a614e5f17063-istio-csr-dns-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:50:19.933356 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.933319 5058 generic.go:356] "Generic (PLEG): container finished" podID="cd5bd6dc-c151-418a-9828-a614e5f17063" containerID="39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e" exitCode=0 Oct 30 04:50:19.933787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.933370 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" Oct 30 04:50:19.933787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.933399 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" event={"ID":"cd5bd6dc-c151-418a-9828-a614e5f17063","Type":"ContainerDied","Data":"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e"} Oct 30 04:50:19.933787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.933452 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf" event={"ID":"cd5bd6dc-c151-418a-9828-a614e5f17063","Type":"ContainerDied","Data":"80f1937bbc9f07f95c1d002126241f03e7f46d66bf6ec6cbce4caabe1264c705"} Oct 30 04:50:19.933787 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.933475 5058 scope.go:122] "RemoveContainer" containerID="39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e" Oct 30 04:50:19.943046 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.943029 5058 scope.go:122] "RemoveContainer" containerID="39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e" Oct 30 04:50:19.943372 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:50:19.943347 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e\": container with ID starting with 39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e not found: ID does not exist" containerID="39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e" Oct 30 04:50:19.943469 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:19.943377 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e"} err="failed to get container status \"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e\": rpc error: code = NotFound desc = could not find container \"39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e\": container with ID starting with 39b4be255278b25bc644e0bac7efdc1513b1bda4aca9c450fe1f4e9fe2ec3b8e not found: ID does not exist" Oct 30 04:50:20.000353 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:20.000316 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:50:20.014010 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:20.013978 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf"] Oct 30 04:50:21.159066 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:50:21.159029 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5bd6dc-c151-418a-9828-a614e5f17063" path="/var/lib/kubelet/pods/cd5bd6dc-c151-418a-9828-a614e5f17063/volumes" Oct 30 04:50:49.190120 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:50:49.190041 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:51:19.190212 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:19.190172 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:51:24.419473 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.419437 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-vsxt8"] Oct 30 04:51:24.421447 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.420686 5058 cpu_manager.go:414] "containerMap: removing container" podUID="cd5bd6dc-c151-418a-9828-a614e5f17063" containerName="discovery" Oct 30 04:51:24.421447 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.420732 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5bd6dc-c151-418a-9828-a614e5f17063" containerName="discovery" Oct 30 04:51:24.421778 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.421754 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5bd6dc-c151-418a-9828-a614e5f17063" containerName="discovery" Oct 30 04:51:24.425952 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.425926 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:24.458705 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.458670 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 04:51:24.458897 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.458878 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 04:51:24.458972 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.458914 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-c8nb2\"" Oct 30 04:51:24.459960 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.459938 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-vsxt8"] Oct 30 04:51:24.569886 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.569837 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxbg\" (UniqueName: \"kubernetes.io/projected/7050e41e-64d0-44ad-bfc8-d1ca9456930b-kube-api-access-6gxbg\") pod \"authorino-operator-68d6dd76dc-vsxt8\" (UID: \"7050e41e-64d0-44ad-bfc8-d1ca9456930b\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:24.670946 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.670861 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxbg\" (UniqueName: \"kubernetes.io/projected/7050e41e-64d0-44ad-bfc8-d1ca9456930b-kube-api-access-6gxbg\") pod \"authorino-operator-68d6dd76dc-vsxt8\" (UID: \"7050e41e-64d0-44ad-bfc8-d1ca9456930b\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:24.694704 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.694667 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxbg\" (UniqueName: \"kubernetes.io/projected/7050e41e-64d0-44ad-bfc8-d1ca9456930b-kube-api-access-6gxbg\") pod \"authorino-operator-68d6dd76dc-vsxt8\" (UID: \"7050e41e-64d0-44ad-bfc8-d1ca9456930b\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:24.739594 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:24.739551 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:25.033662 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:25.033632 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-vsxt8"] Oct 30 04:51:25.034726 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:51:25.034695 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7050e41e_64d0_44ad_bfc8_d1ca9456930b.slice/crio-1b6bf95c5d548a97b098cda70ddabc6cb245a2b323310f758e3898899e553e4d WatchSource:0}: Error finding container 1b6bf95c5d548a97b098cda70ddabc6cb245a2b323310f758e3898899e553e4d: Status 404 returned error can't find the container with id 1b6bf95c5d548a97b098cda70ddabc6cb245a2b323310f758e3898899e553e4d Oct 30 04:51:25.205954 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:25.205915 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" event={"ID":"7050e41e-64d0-44ad-bfc8-d1ca9456930b","Type":"ContainerStarted","Data":"1b6bf95c5d548a97b098cda70ddabc6cb245a2b323310f758e3898899e553e4d"} Oct 30 04:51:26.710103 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.710066 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz"] Oct 30 04:51:26.718790 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.718765 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:26.737307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.737273 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz"] Oct 30 04:51:26.748685 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.748654 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jrx5s\"" Oct 30 04:51:26.748685 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.748654 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 04:51:26.889736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.889695 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7lp\" (UniqueName: \"kubernetes.io/projected/2c2154fe-808a-4775-b983-8dee7295a515-kube-api-access-hw7lp\") pod \"dns-operator-controller-manager-65bfd79598-9s8jz\" (UID: \"2c2154fe-808a-4775-b983-8dee7295a515\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:26.990579 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:26.990489 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7lp\" (UniqueName: \"kubernetes.io/projected/2c2154fe-808a-4775-b983-8dee7295a515-kube-api-access-hw7lp\") pod \"dns-operator-controller-manager-65bfd79598-9s8jz\" (UID: \"2c2154fe-808a-4775-b983-8dee7295a515\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:27.016403 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:27.016337 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7lp\" (UniqueName: \"kubernetes.io/projected/2c2154fe-808a-4775-b983-8dee7295a515-kube-api-access-hw7lp\") pod \"dns-operator-controller-manager-65bfd79598-9s8jz\" (UID: \"2c2154fe-808a-4775-b983-8dee7295a515\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:27.034263 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:27.034201 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:27.336815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:27.336727 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz"] Oct 30 04:51:27.345701 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:51:27.345668 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2154fe_808a_4775_b983_8dee7295a515.slice/crio-1ccb1cbd9123aa2fa3952db3b95916ee5d9b05d152569c7e02962a0a26096390 WatchSource:0}: Error finding container 1ccb1cbd9123aa2fa3952db3b95916ee5d9b05d152569c7e02962a0a26096390: Status 404 returned error can't find the container with id 1ccb1cbd9123aa2fa3952db3b95916ee5d9b05d152569c7e02962a0a26096390 Oct 30 04:51:28.223061 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:28.223024 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" event={"ID":"2c2154fe-808a-4775-b983-8dee7295a515","Type":"ContainerStarted","Data":"1ccb1cbd9123aa2fa3952db3b95916ee5d9b05d152569c7e02962a0a26096390"} Oct 30 04:51:28.224829 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:28.224796 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" event={"ID":"7050e41e-64d0-44ad-bfc8-d1ca9456930b","Type":"ContainerStarted","Data":"f9c47eb5ab02be15cb5262db137f46da89c04df42e0ca398db06b40063c5b12f"} Oct 30 04:51:28.224972 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:28.224951 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:28.265982 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:28.265920 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" podStartSLOduration=1.7363732440000001 podStartE2EDuration="4.265905976s" totalImagesPullingTime="2.529532732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:28.264590988 +0000 UTC m=+461.234085978" watchObservedRunningTime="2026-10-30 04:51:28.265905976 +0000 UTC m=+461.235400998" Oct 30 04:51:29.230974 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.230935 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" event={"ID":"2c2154fe-808a-4775-b983-8dee7295a515","Type":"ContainerStarted","Data":"123ab7aa15d426497523618e59cb0d8cc7c9c9bc5aaefc7afd3e36ae9160861e"} Oct 30 04:51:29.231386 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.231157 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:29.235636 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.235611 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6"] Oct 30 04:51:29.238925 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.238908 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:29.265232 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.265199 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nd268\"" Oct 30 04:51:29.267430 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.267403 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6"] Oct 30 04:51:29.283030 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.282973 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" podStartSLOduration=1.882364225 podStartE2EDuration="3.282958996s" totalImagesPullingTime="1.400594771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:29.282163204 +0000 UTC m=+462.251658193" watchObservedRunningTime="2026-10-30 04:51:29.282958996 +0000 UTC m=+462.252453990" Oct 30 04:51:29.311070 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.311034 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwwp\" (UniqueName: \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\") pod \"limitador-operator-controller-manager-856dd584c5-m9sl6\" (UID: \"771d7221-6052-4c57-ad88-e32cf3018324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:29.412506 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.412468 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwwp\" (UniqueName: \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\") pod \"limitador-operator-controller-manager-856dd584c5-m9sl6\" (UID: \"771d7221-6052-4c57-ad88-e32cf3018324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:29.442032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.441997 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwwp\" (UniqueName: \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\") pod \"limitador-operator-controller-manager-856dd584c5-m9sl6\" (UID: \"771d7221-6052-4c57-ad88-e32cf3018324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:29.551409 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.551324 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:29.848008 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:29.847979 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6"] Oct 30 04:51:29.849313 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:51:29.849285 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771d7221_6052_4c57_ad88_e32cf3018324.slice/crio-48d8f8ce78176bda42a074f056b6e0555e5a4df603afc8e49c286a63a4d1c816 WatchSource:0}: Error finding container 48d8f8ce78176bda42a074f056b6e0555e5a4df603afc8e49c286a63a4d1c816: Status 404 returned error can't find the container with id 48d8f8ce78176bda42a074f056b6e0555e5a4df603afc8e49c286a63a4d1c816 Oct 30 04:51:30.239199 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:30.239162 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" event={"ID":"771d7221-6052-4c57-ad88-e32cf3018324","Type":"ContainerStarted","Data":"48d8f8ce78176bda42a074f056b6e0555e5a4df603afc8e49c286a63a4d1c816"} Oct 30 04:51:33.253798 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:33.253765 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" event={"ID":"771d7221-6052-4c57-ad88-e32cf3018324","Type":"ContainerStarted","Data":"59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0"} Oct 30 04:51:33.254234 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:33.253889 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:33.297886 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:33.297830 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" podStartSLOduration=1.883308948 podStartE2EDuration="4.297814372s" totalImagesPullingTime="2.414505424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:33.295929243 +0000 UTC m=+466.265424242" watchObservedRunningTime="2026-10-30 04:51:33.297814372 +0000 UTC m=+466.267309361" Oct 30 04:51:39.233512 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:39.233480 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-vsxt8" Oct 30 04:51:40.242250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:40.242202 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9s8jz" Oct 30 04:51:44.261058 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:44.261025 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:49.190440 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:49.190403 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:51:52.786318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.786273 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6"] Oct 30 04:51:52.786717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.786496 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" podUID="771d7221-6052-4c57-ad88-e32cf3018324" containerName="manager" containerID="cri-o://59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0" gracePeriod=2 Oct 30 04:51:52.800122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.800092 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6"] Oct 30 04:51:52.800970 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:52.800939 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-m9sl6\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="771d7221-6052-4c57-ad88-e32cf3018324" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:52.810843 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.810814 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r"] Oct 30 04:51:52.811193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.811178 5058 cpu_manager.go:414] "containerMap: removing container" podUID="771d7221-6052-4c57-ad88-e32cf3018324" containerName="manager" Oct 30 04:51:52.811193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.811191 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771d7221-6052-4c57-ad88-e32cf3018324" containerName="manager" Oct 30 04:51:52.811363 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.811300 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="771d7221-6052-4c57-ad88-e32cf3018324" containerName="manager" Oct 30 04:51:52.814453 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.814429 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:52.829982 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:52.829946 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-m9sl6\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="771d7221-6052-4c57-ad88-e32cf3018324" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:52.839768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.839739 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r"] Oct 30 04:51:52.903795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:52.903761 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7znrh\" (UniqueName: \"kubernetes.io/projected/0f58bc18-62b8-46a7-b845-d91cb7d936e1-kube-api-access-7znrh\") pod \"limitador-operator-controller-manager-856dd584c5-n4k9r\" (UID: \"0f58bc18-62b8-46a7-b845-d91cb7d936e1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:53.004783 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.004751 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7znrh\" (UniqueName: \"kubernetes.io/projected/0f58bc18-62b8-46a7-b845-d91cb7d936e1-kube-api-access-7znrh\") pod \"limitador-operator-controller-manager-856dd584c5-n4k9r\" (UID: \"0f58bc18-62b8-46a7-b845-d91cb7d936e1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:53.033869 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.033843 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7znrh\" (UniqueName: \"kubernetes.io/projected/0f58bc18-62b8-46a7-b845-d91cb7d936e1-kube-api-access-7znrh\") pod \"limitador-operator-controller-manager-856dd584c5-n4k9r\" (UID: \"0f58bc18-62b8-46a7-b845-d91cb7d936e1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:53.052617 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.052559 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:53.067191 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:53.067143 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-m9sl6\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="771d7221-6052-4c57-ad88-e32cf3018324" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:53.105858 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.105822 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\" (UniqueName: \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\") pod \"771d7221-6052-4c57-ad88-e32cf3018324\" (UID: \"771d7221-6052-4c57-ad88-e32cf3018324\") " Oct 30 04:51:53.108069 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.108040 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp" pod "771d7221-6052-4c57-ad88-e32cf3018324" (UID: "771d7221-6052-4c57-ad88-e32cf3018324"). InnerVolumeSpecName "kube-api-access-htwwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:53.158936 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.158908 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771d7221-6052-4c57-ad88-e32cf3018324" path="/var/lib/kubelet/pods/771d7221-6052-4c57-ad88-e32cf3018324/volumes" Oct 30 04:51:53.178507 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.178484 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:53.206730 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.206700 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwwp\" (UniqueName: \"kubernetes.io/projected/771d7221-6052-4c57-ad88-e32cf3018324-kube-api-access-htwwp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:51:53.343912 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.343830 5058 generic.go:356] "Generic (PLEG): container finished" podID="771d7221-6052-4c57-ad88-e32cf3018324" containerID="59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0" exitCode=0 Oct 30 04:51:53.343912 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.343876 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m9sl6" Oct 30 04:51:53.343912 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.343897 5058 scope.go:122] "RemoveContainer" containerID="59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0" Oct 30 04:51:53.352354 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.352331 5058 scope.go:122] "RemoveContainer" containerID="59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0" Oct 30 04:51:53.352622 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:51:53.352600 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0\": container with ID starting with 59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0 not found: ID does not exist" containerID="59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0" Oct 30 04:51:53.352672 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.352633 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0"} err="failed to get container status \"59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0\": rpc error: code = NotFound desc = could not find container \"59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0\": container with ID starting with 59ace5a8a9d90c316e0cbefeddb66fa64ff82774bf56632ade49e1843c1fefd0 not found: ID does not exist" Oct 30 04:51:53.468681 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:53.468656 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r"] Oct 30 04:51:53.469591 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:51:53.469548 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f58bc18_62b8_46a7_b845_d91cb7d936e1.slice/crio-4fafad7ddfa7e3266d939e02bd880a5084bc8bfaecd62383c8455c752e284448 WatchSource:0}: Error finding container 4fafad7ddfa7e3266d939e02bd880a5084bc8bfaecd62383c8455c752e284448: Status 404 returned error can't find the container with id 4fafad7ddfa7e3266d939e02bd880a5084bc8bfaecd62383c8455c752e284448 Oct 30 04:51:54.351770 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:54.351738 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" event={"ID":"0f58bc18-62b8-46a7-b845-d91cb7d936e1","Type":"ContainerStarted","Data":"2e8a9ed67dfaa67dd2e2db9f73bea8dc660c4a07657f3e1f27b0771bb8faa77d"} Oct 30 04:51:54.352174 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:54.351801 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:51:54.352174 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:54.351824 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" event={"ID":"0f58bc18-62b8-46a7-b845-d91cb7d936e1","Type":"ContainerStarted","Data":"4fafad7ddfa7e3266d939e02bd880a5084bc8bfaecd62383c8455c752e284448"} Oct 30 04:51:54.395351 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:51:54.395300 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" podStartSLOduration=2.39528463 podStartE2EDuration="2.39528463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:54.393579958 +0000 UTC m=+487.363074929" watchObservedRunningTime="2026-10-30 04:51:54.39528463 +0000 UTC m=+487.364779619" Oct 30 04:52:05.358063 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:05.358030 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n4k9r" Oct 30 04:52:19.189642 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:52:19.189601 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:52:40.564992 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.564936 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:40.568755 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.568731 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.585470 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.585443 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:40.597598 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.597545 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 04:52:40.597598 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.597594 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c8tcm\"" Oct 30 04:52:40.627159 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.627124 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.627370 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.627245 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxqn\" (UniqueName: \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.700793 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.700762 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:40.728427 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.728395 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.728660 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.728486 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxqn\" (UniqueName: \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.729060 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.729041 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.748630 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.748596 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxqn\" (UniqueName: \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\") pod \"limitador-limitador-74dfcfb894-gzzp4\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:40.881931 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:40.881839 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:41.174347 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:41.174321 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:41.176144 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:52:41.176119 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7860b7d_c8ee_41df_93ff_558831e3ae6c.slice/crio-034a24f6ea7780c7c18895b2e5fa03055844c58579102bacea8dc1239d662ac3 WatchSource:0}: Error finding container 034a24f6ea7780c7c18895b2e5fa03055844c58579102bacea8dc1239d662ac3: Status 404 returned error can't find the container with id 034a24f6ea7780c7c18895b2e5fa03055844c58579102bacea8dc1239d662ac3 Oct 30 04:52:41.561832 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:41.561756 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" event={"ID":"b7860b7d-c8ee-41df-93ff-558831e3ae6c","Type":"ContainerStarted","Data":"034a24f6ea7780c7c18895b2e5fa03055844c58579102bacea8dc1239d662ac3"} Oct 30 04:52:42.567642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:42.567544 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" event={"ID":"b7860b7d-c8ee-41df-93ff-558831e3ae6c","Type":"ContainerStarted","Data":"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4"} Oct 30 04:52:42.567642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:42.567624 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:42.608601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:42.608543 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" podStartSLOduration=1.4976023889999999 podStartE2EDuration="2.608527887s" totalImagesPullingTime="1.110925498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:42.607313064 +0000 UTC m=+535.576808052" watchObservedRunningTime="2026-10-30 04:52:42.608527887 +0000 UTC m=+535.578022876" Oct 30 04:52:49.190925 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:52:49.190877 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:52:53.573062 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:53.573025 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:55.006838 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.006800 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:55.007265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.007025 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" podUID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" containerName="limitador" containerID="cri-o://8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4" gracePeriod=30 Oct 30 04:52:55.562594 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.562569 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:55.629062 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.628973 5058 generic.go:356] "Generic (PLEG): container finished" podID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" containerID="8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4" exitCode=0 Oct 30 04:52:55.629062 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.629026 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" Oct 30 04:52:55.629326 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.629058 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" event={"ID":"b7860b7d-c8ee-41df-93ff-558831e3ae6c","Type":"ContainerDied","Data":"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4"} Oct 30 04:52:55.629326 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.629122 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gzzp4" event={"ID":"b7860b7d-c8ee-41df-93ff-558831e3ae6c","Type":"ContainerDied","Data":"034a24f6ea7780c7c18895b2e5fa03055844c58579102bacea8dc1239d662ac3"} Oct 30 04:52:55.629326 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.629154 5058 scope.go:122] "RemoveContainer" containerID="8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4" Oct 30 04:52:55.638774 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.638751 5058 scope.go:122] "RemoveContainer" containerID="8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4" Oct 30 04:52:55.639109 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:52:55.639088 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4\": container with ID starting with 8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4 not found: ID does not exist" containerID="8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4" Oct 30 04:52:55.639190 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.639122 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4"} err="failed to get container status \"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4\": rpc error: code = NotFound desc = could not find container \"8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4\": container with ID starting with 8c4c48b1f0c8fe428ce4e9b6b38865c89556f82f04412f395cfff8fdf3116dc4 not found: ID does not exist" Oct 30 04:52:55.644555 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.644533 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\" (UniqueName: \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\") pod \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " Oct 30 04:52:55.644669 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.644653 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\" (UniqueName: \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\") pod \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\" (UID: \"b7860b7d-c8ee-41df-93ff-558831e3ae6c\") " Oct 30 04:52:55.644928 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.644908 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file" pod "b7860b7d-c8ee-41df-93ff-558831e3ae6c" (UID: "b7860b7d-c8ee-41df-93ff-558831e3ae6c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:52:55.646928 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.646899 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn" pod "b7860b7d-c8ee-41df-93ff-558831e3ae6c" (UID: "b7860b7d-c8ee-41df-93ff-558831e3ae6c"). InnerVolumeSpecName "kube-api-access-zlxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:55.746184 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.746145 5058 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7860b7d-c8ee-41df-93ff-558831e3ae6c-config-file\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:52:55.746184 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.746181 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxqn\" (UniqueName: \"kubernetes.io/projected/b7860b7d-c8ee-41df-93ff-558831e3ae6c-kube-api-access-zlxqn\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:52:55.976170 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.976125 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:55.989891 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:55.989852 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gzzp4"] Oct 30 04:52:57.158772 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:52:57.158737 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" path="/var/lib/kubelet/pods/b7860b7d-c8ee-41df-93ff-558831e3ae6c/volumes" Oct 30 04:53:01.644554 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.644515 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj"] Oct 30 04:53:01.644943 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.644891 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" containerName="limitador" Oct 30 04:53:01.644943 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.644901 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" containerName="limitador" Oct 30 04:53:01.645017 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.644975 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7860b7d-c8ee-41df-93ff-558831e3ae6c" containerName="limitador" Oct 30 04:53:01.648126 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.648103 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.664210 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.664182 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:53:01.664431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.664242 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:53:01.665670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.665649 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj"] Oct 30 04:53:01.678055 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.678024 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d77n4\"" Oct 30 04:53:01.678198 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.678070 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 04:53:01.703236 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.703180 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrzb\" (UniqueName: \"kubernetes.io/projected/21ca24c3-36fe-40f9-90a2-ef416baa6167-kube-api-access-zrrzb\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.703399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.703284 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21ca24c3-36fe-40f9-90a2-ef416baa6167-data\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.803795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.803758 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21ca24c3-36fe-40f9-90a2-ef416baa6167-data\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.803969 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.803857 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrzb\" (UniqueName: \"kubernetes.io/projected/21ca24c3-36fe-40f9-90a2-ef416baa6167-kube-api-access-zrrzb\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.804273 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.804254 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21ca24c3-36fe-40f9-90a2-ef416baa6167-data\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.825717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.825687 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrzb\" (UniqueName: \"kubernetes.io/projected/21ca24c3-36fe-40f9-90a2-ef416baa6167-kube-api-access-zrrzb\") pod \"postgres-7cdbf6dc44-nl2nj\" (UID: \"21ca24c3-36fe-40f9-90a2-ef416baa6167\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:01.960716 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:01.960624 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:02.250480 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:02.250446 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj"] Oct 30 04:53:02.251146 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:53:02.251120 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ca24c3_36fe_40f9_90a2_ef416baa6167.slice/crio-a5320a9deca10107d5fc4acf5207d5f2f3cee06c166010838c23667182383c45 WatchSource:0}: Error finding container a5320a9deca10107d5fc4acf5207d5f2f3cee06c166010838c23667182383c45: Status 404 returned error can't find the container with id a5320a9deca10107d5fc4acf5207d5f2f3cee06c166010838c23667182383c45 Oct 30 04:53:02.663079 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:02.663042 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" event={"ID":"21ca24c3-36fe-40f9-90a2-ef416baa6167","Type":"ContainerStarted","Data":"a5320a9deca10107d5fc4acf5207d5f2f3cee06c166010838c23667182383c45"} Oct 30 04:53:07.690749 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:07.690709 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" event={"ID":"21ca24c3-36fe-40f9-90a2-ef416baa6167","Type":"ContainerStarted","Data":"93eda389c568012a2671936190e3bed32ccf87f2cb6072d4a4899f2ba0b111ee"} Oct 30 04:53:07.691141 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:07.690837 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:07.730797 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:07.730748 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" podStartSLOduration=1.813055904 podStartE2EDuration="6.730734756s" totalImagesPullingTime="4.917678852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:07.73013712 +0000 UTC m=+560.699632109" watchObservedRunningTime="2026-10-30 04:53:07.730734756 +0000 UTC m=+560.700229753" Oct 30 04:53:13.724870 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:13.724835 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nl2nj" Oct 30 04:53:19.189254 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:53:19.189122 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:53:23.671439 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.671401 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:23.675104 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.675082 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.704268 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.704207 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:23.705376 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.705353 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 04:53:23.724683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.724647 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 04:53:23.724683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.724660 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 04:53:23.724887 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.724709 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c5w2d\"" Oct 30 04:53:23.808079 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.808031 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bjl\" (UniqueName: \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.808286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.808101 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.808286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.808195 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.909656 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.909616 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.909656 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.909660 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.909860 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.909714 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bjl\" (UniqueName: \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.912143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.912110 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.912289 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.912164 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.930145 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.930080 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bjl\" (UniqueName: \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\") pod \"maas-controller-577f596567-5pmj8\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:23.989136 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:23.989092 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:24.049401 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.049366 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 04:53:24.056904 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.056873 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.072307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.072280 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 04:53:24.212788 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.212705 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.212788 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.212753 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmc5\" (UniqueName: \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.212970 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.212825 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.281178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.281151 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:24.282960 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:53:24.282928 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cdce22_6f4c_47aa_a745_7e441347ddb6.slice/crio-9d2d1e5c5578cc3ea556b0bf5c047c50d7fd653591ccc37bb372f1bdb4a53b0d WatchSource:0}: Error finding container 9d2d1e5c5578cc3ea556b0bf5c047c50d7fd653591ccc37bb372f1bdb4a53b0d: Status 404 returned error can't find the container with id 9d2d1e5c5578cc3ea556b0bf5c047c50d7fd653591ccc37bb372f1bdb4a53b0d Oct 30 04:53:24.314100 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.314067 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.314307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.314143 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.314307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.314178 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmc5\" (UniqueName: \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.316543 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.316514 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.316654 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.316546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.334135 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.334109 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmc5\" (UniqueName: \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\") pod \"maas-controller-5d77b88d59-8t5sz\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.375284 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.375253 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:24.665388 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.665355 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 04:53:24.666328 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:53:24.666301 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3661465d_75ee_4b89_8e02_5dfb791997f0.slice/crio-7a652bbbcc5c0727856e05156e4e6770d335466fca504657a1ddd8fe1c2da4b9 WatchSource:0}: Error finding container 7a652bbbcc5c0727856e05156e4e6770d335466fca504657a1ddd8fe1c2da4b9: Status 404 returned error can't find the container with id 7a652bbbcc5c0727856e05156e4e6770d335466fca504657a1ddd8fe1c2da4b9 Oct 30 04:53:24.771192 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.771154 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5pmj8" event={"ID":"e0cdce22-6f4c-47aa-a745-7e441347ddb6","Type":"ContainerStarted","Data":"9d2d1e5c5578cc3ea556b0bf5c047c50d7fd653591ccc37bb372f1bdb4a53b0d"} Oct 30 04:53:24.772330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:24.772304 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" event={"ID":"3661465d-75ee-4b89-8e02-5dfb791997f0","Type":"ContainerStarted","Data":"7a652bbbcc5c0727856e05156e4e6770d335466fca504657a1ddd8fe1c2da4b9"} Oct 30 04:53:27.790749 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.790666 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5pmj8" event={"ID":"e0cdce22-6f4c-47aa-a745-7e441347ddb6","Type":"ContainerStarted","Data":"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62"} Oct 30 04:53:27.791229 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.790798 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:27.792301 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.792277 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" event={"ID":"3661465d-75ee-4b89-8e02-5dfb791997f0","Type":"ContainerStarted","Data":"41bd7fbb49fa20e886313a521ac83f339a162dceba47ff55b76c3c317051643c"} Oct 30 04:53:27.792405 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.792321 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:27.833692 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.833641 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-5pmj8" podStartSLOduration=2.070641106 podStartE2EDuration="4.833626877s" totalImagesPullingTime="2.762985771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:27.832193987 +0000 UTC m=+580.801688981" watchObservedRunningTime="2026-10-30 04:53:27.833626877 +0000 UTC m=+580.803121866" Oct 30 04:53:27.875415 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:27.875364 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" podStartSLOduration=1.183961194 podStartE2EDuration="3.875354303s" totalImagesPullingTime="2.691393109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:27.874654574 +0000 UTC m=+580.844149562" watchObservedRunningTime="2026-10-30 04:53:27.875354303 +0000 UTC m=+580.844849292" Oct 30 04:53:38.799147 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:38.799112 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 04:53:38.799647 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:38.799202 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:38.913737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:38.913699 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:38.913970 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:38.913944 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-5pmj8" podUID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" containerName="manager" containerID="cri-o://bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62" gracePeriod=30 Oct 30 04:53:39.160201 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.160177 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:39.239514 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.239485 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\") pod \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " Oct 30 04:53:39.239684 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.239525 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\") pod \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " Oct 30 04:53:39.239684 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.239570 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\" (UniqueName: \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\") pod \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\" (UID: \"e0cdce22-6f4c-47aa-a745-7e441347ddb6\") " Oct 30 04:53:39.241733 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.241702 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert" pod "e0cdce22-6f4c-47aa-a745-7e441347ddb6" (UID: "e0cdce22-6f4c-47aa-a745-7e441347ddb6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:53:39.241878 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.241747 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs" pod "e0cdce22-6f4c-47aa-a745-7e441347ddb6" (UID: "e0cdce22-6f4c-47aa-a745-7e441347ddb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:53:39.241878 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.241834 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl" pod "e0cdce22-6f4c-47aa-a745-7e441347ddb6" (UID: "e0cdce22-6f4c-47aa-a745-7e441347ddb6"). InnerVolumeSpecName "kube-api-access-v8bjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:53:39.340861 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.340776 5058 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-webhook-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:53:39.340861 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.340809 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cdce22-6f4c-47aa-a745-7e441347ddb6-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:53:39.340861 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.340819 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bjl\" (UniqueName: \"kubernetes.io/projected/e0cdce22-6f4c-47aa-a745-7e441347ddb6-kube-api-access-v8bjl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 04:53:39.848815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.848779 5058 generic.go:356] "Generic (PLEG): container finished" podID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" containerID="bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62" exitCode=0 Oct 30 04:53:39.849259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.848829 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5pmj8" Oct 30 04:53:39.849259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.848862 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5pmj8" event={"ID":"e0cdce22-6f4c-47aa-a745-7e441347ddb6","Type":"ContainerDied","Data":"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62"} Oct 30 04:53:39.849259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.848917 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5pmj8" event={"ID":"e0cdce22-6f4c-47aa-a745-7e441347ddb6","Type":"ContainerDied","Data":"9d2d1e5c5578cc3ea556b0bf5c047c50d7fd653591ccc37bb372f1bdb4a53b0d"} Oct 30 04:53:39.849259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.848945 5058 scope.go:122] "RemoveContainer" containerID="bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62" Oct 30 04:53:39.858717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.858699 5058 scope.go:122] "RemoveContainer" containerID="bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62" Oct 30 04:53:39.858973 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:53:39.858951 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62\": container with ID starting with bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62 not found: ID does not exist" containerID="bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62" Oct 30 04:53:39.859031 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.858982 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62"} err="failed to get container status \"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62\": rpc error: code = NotFound desc = could not find container \"bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62\": container with ID starting with bb2978d60ca5eef74e002d9725f8a0f6578b6a3d7e40fee15cd43d0024ea4d62 not found: ID does not exist" Oct 30 04:53:39.902442 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.902410 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:39.916062 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:39.916024 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-5pmj8"] Oct 30 04:53:41.158815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:41.158784 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" path="/var/lib/kubelet/pods/e0cdce22-6f4c-47aa-a745-7e441347ddb6/volumes" Oct 30 04:53:47.222674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:47.222642 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:53:47.225191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:53:47.225166 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:53:49.189786 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:53:49.189745 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:54:01.686172 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.686136 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 04:54:01.686666 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.686559 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" containerName="manager" Oct 30 04:54:01.686666 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.686572 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" containerName="manager" Oct 30 04:54:01.686666 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.686658 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cdce22-6f4c-47aa-a745-7e441347ddb6" containerName="manager" Oct 30 04:54:01.689582 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.689564 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.708696 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.708667 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 04:54:01.720321 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.720286 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 04:54:01.720475 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.720342 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-grc2l\"" Oct 30 04:54:01.723082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.723061 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 04:54:01.842553 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.842520 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.842744 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.842564 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.842744 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.842583 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.842744 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.842604 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56pp\" (UniqueName: \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.943643 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.943559 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.943643 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.943598 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.943643 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.943620 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56pp\" (UniqueName: \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.943865 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.943800 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.944397 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.944367 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.946130 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.946104 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.946212 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.946159 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:01.963900 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:01.963872 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56pp\" (UniqueName: \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\") pod \"maas-api-5c48877c75-rxlxx\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:02.001863 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:02.001825 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:02.293021 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:02.292989 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 04:54:02.293727 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:54:02.293700 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6f1fa4_4bd9_4a72_b4ee_a354e6e0f9bb.slice/crio-f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2 WatchSource:0}: Error finding container f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2: Status 404 returned error can't find the container with id f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2 Oct 30 04:54:02.294965 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:02.294945 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:54:02.968663 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:02.968629 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" event={"ID":"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb","Type":"ContainerStarted","Data":"f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2"} Oct 30 04:54:04.981956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:04.981913 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" event={"ID":"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb","Type":"ContainerStarted","Data":"561ad18bc76be8101862894761f11582a75f7093c3df6bf9bef9bade84ef3c1e"} Oct 30 04:54:04.982382 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:04.982027 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:05.023752 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:05.023706 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" podStartSLOduration=2.369015994 podStartE2EDuration="4.023692849s" totalImagesPullingTime="1.654676855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:05.022390516 +0000 UTC m=+617.991885505" watchObservedRunningTime="2026-10-30 04:54:05.023692849 +0000 UTC m=+617.993187838" Oct 30 04:54:10.991075 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:54:10.991047 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 04:54:19.193209 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:54:19.193166 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:54:49.190392 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:54:49.190357 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:55:04.830495 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.830455 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455"] Oct 30 04:55:04.834214 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.834193 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.866342 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.866304 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5wsj\"" Oct 30 04:55:04.866515 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.866412 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 04:55:04.867891 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.867871 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455"] Oct 30 04:55:04.905330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.905297 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 04:55:04.905330 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.905297 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 04:55:04.991610 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991570 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.991610 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991613 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8djw\" (UniqueName: \"kubernetes.io/projected/b292f152-c018-4589-a01b-e7f49c505f47-kube-api-access-b8djw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.991830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991728 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.991830 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991769 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292f152-c018-4589-a01b-e7f49c505f47-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.991928 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991845 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:04.991976 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:04.991927 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093438 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093345 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292f152-c018-4589-a01b-e7f49c505f47-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093438 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093402 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093690 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093440 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093690 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093478 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093690 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093500 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8djw\" (UniqueName: \"kubernetes.io/projected/b292f152-c018-4589-a01b-e7f49c505f47-kube-api-access-b8djw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093690 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093575 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093898 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093862 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.093956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093897 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.094023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.093966 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.095815 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.095793 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b292f152-c018-4589-a01b-e7f49c505f47-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.096054 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.096038 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292f152-c018-4589-a01b-e7f49c505f47-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.116412 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.116375 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8djw\" (UniqueName: \"kubernetes.io/projected/b292f152-c018-4589-a01b-e7f49c505f47-kube-api-access-b8djw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455\" (UID: \"b292f152-c018-4589-a01b-e7f49c505f47\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.149964 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.149923 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:05.441264 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:55:05.441230 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb292f152_c018_4589_a01b_e7f49c505f47.slice/crio-87538dfa1378cdebabd3662c11b45995c3a4ec9331143b0f6a06147018badeb6 WatchSource:0}: Error finding container 87538dfa1378cdebabd3662c11b45995c3a4ec9331143b0f6a06147018badeb6: Status 404 returned error can't find the container with id 87538dfa1378cdebabd3662c11b45995c3a4ec9331143b0f6a06147018badeb6 Oct 30 04:55:05.441436 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:05.441413 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455"] Oct 30 04:55:06.266618 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:06.266581 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" event={"ID":"b292f152-c018-4589-a01b-e7f49c505f47","Type":"ContainerStarted","Data":"87538dfa1378cdebabd3662c11b45995c3a4ec9331143b0f6a06147018badeb6"} Oct 30 04:55:07.273208 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:07.273179 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" event={"ID":"b292f152-c018-4589-a01b-e7f49c505f47","Type":"ContainerStarted","Data":"b3e4ba43030d982554823af82cf727e10e8381e785786404c2e1c89727b3505c"} Oct 30 04:55:07.315750 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:07.315695 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" podStartSLOduration=2.243938448 podStartE2EDuration="3.315682167s" totalImagesPullingTime="1.071743719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:07.314141793 +0000 UTC m=+680.283636786" watchObservedRunningTime="2026-10-30 04:55:07.315682167 +0000 UTC m=+680.285177155" Oct 30 04:55:08.150427 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:08.150391 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:10.012577 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.012537 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh"] Oct 30 04:55:10.016340 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.016318 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.037794 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.037762 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh"] Oct 30 04:55:10.048628 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.048605 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 04:55:10.139004 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.138976 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.139149 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.139024 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab64951-a948-4e61-9a3c-58438e9dadd5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.139149 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.139133 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.139243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.139163 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.139243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.139180 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.139307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.139248 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szd42\" (UniqueName: \"kubernetes.io/projected/2ab64951-a948-4e61-9a3c-58438e9dadd5-kube-api-access-szd42\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.239952 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.239912 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.239952 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.239946 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240187 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.239975 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szd42\" (UniqueName: \"kubernetes.io/projected/2ab64951-a948-4e61-9a3c-58438e9dadd5-kube-api-access-szd42\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240187 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240049 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240335 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240202 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab64951-a948-4e61-9a3c-58438e9dadd5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240386 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240342 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240443 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240392 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240734 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240708 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.240867 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.240736 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.242426 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.242407 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ab64951-a948-4e61-9a3c-58438e9dadd5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.242693 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.242677 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab64951-a948-4e61-9a3c-58438e9dadd5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.259905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.259882 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szd42\" (UniqueName: \"kubernetes.io/projected/2ab64951-a948-4e61-9a3c-58438e9dadd5-kube-api-access-szd42\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh\" (UID: \"2ab64951-a948-4e61-9a3c-58438e9dadd5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.331573 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.331494 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:10.622049 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:10.622018 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh"] Oct 30 04:55:10.622194 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:55:10.622167 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab64951_a948_4e61_9a3c_58438e9dadd5.slice/crio-c475368f8d0de1a769c580deba35793dd117a04d98ef2ad96896ce7194eb2d66 WatchSource:0}: Error finding container c475368f8d0de1a769c580deba35793dd117a04d98ef2ad96896ce7194eb2d66: Status 404 returned error can't find the container with id c475368f8d0de1a769c580deba35793dd117a04d98ef2ad96896ce7194eb2d66 Oct 30 04:55:11.293362 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.293329 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" event={"ID":"2ab64951-a948-4e61-9a3c-58438e9dadd5","Type":"ContainerStarted","Data":"90a8f26c87d81b189830ece5072501e93ba0235f8156c9c62af354bc335500e3"} Oct 30 04:55:11.293751 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.293381 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" event={"ID":"2ab64951-a948-4e61-9a3c-58438e9dadd5","Type":"ContainerStarted","Data":"c475368f8d0de1a769c580deba35793dd117a04d98ef2ad96896ce7194eb2d66"} Oct 30 04:55:11.331891 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.331865 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:11.334413 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.334371 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" podStartSLOduration=2.132334132 podStartE2EDuration="2.334355607s" totalImagesPullingTime="202.021475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:11.333615518 +0000 UTC m=+684.303110508" watchObservedRunningTime="2026-10-30 04:55:11.334355607 +0000 UTC m=+684.303850596" Oct 30 04:55:11.840416 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.840381 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d"] Oct 30 04:55:11.844274 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.844201 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.863391 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.863362 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d"] Oct 30 04:55:11.874521 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.874496 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 04:55:11.956577 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956532 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.956577 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956570 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.956840 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956594 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.956840 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956697 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.956840 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956809 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:11.956997 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:11.956849 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgjc\" (UniqueName: \"kubernetes.io/projected/5409a084-5c7a-4ad4-87db-98dfe91eadc3-kube-api-access-chgjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057711 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057669 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057711 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057715 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057938 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057738 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057938 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057786 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057938 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057834 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.057938 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.057886 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgjc\" (UniqueName: \"kubernetes.io/projected/5409a084-5c7a-4ad4-87db-98dfe91eadc3-kube-api-access-chgjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.058137 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.058083 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.058194 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.058145 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.058289 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.058270 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.060040 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.060019 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409a084-5c7a-4ad4-87db-98dfe91eadc3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.060322 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.060304 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409a084-5c7a-4ad4-87db-98dfe91eadc3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.077911 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.077879 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgjc\" (UniqueName: \"kubernetes.io/projected/5409a084-5c7a-4ad4-87db-98dfe91eadc3-kube-api-access-chgjc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-snw9d\" (UID: \"5409a084-5c7a-4ad4-87db-98dfe91eadc3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.158987 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.158948 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:12.450118 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:12.450089 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d"] Oct 30 04:55:12.450684 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:55:12.450661 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5409a084_5c7a_4ad4_87db_98dfe91eadc3.slice/crio-1b96d777ad420cc9cf33fe46ad6f1030690d50a2a67d09e8a06233d1f879d845 WatchSource:0}: Error finding container 1b96d777ad420cc9cf33fe46ad6f1030690d50a2a67d09e8a06233d1f879d845: Status 404 returned error can't find the container with id 1b96d777ad420cc9cf33fe46ad6f1030690d50a2a67d09e8a06233d1f879d845 Oct 30 04:55:13.305519 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:13.305483 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" event={"ID":"5409a084-5c7a-4ad4-87db-98dfe91eadc3","Type":"ContainerStarted","Data":"c36fcd8d411a01f52f07b311819cc2773bf9f89f6fa7f74ed59ea643fd72a66a"} Oct 30 04:55:13.305700 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:13.305556 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" event={"ID":"5409a084-5c7a-4ad4-87db-98dfe91eadc3","Type":"ContainerStarted","Data":"1b96d777ad420cc9cf33fe46ad6f1030690d50a2a67d09e8a06233d1f879d845"} Oct 30 04:55:13.348074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:13.348020 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" podStartSLOduration=2.131394105 podStartE2EDuration="2.348007565s" totalImagesPullingTime="216.61346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:13.346604975 +0000 UTC m=+686.316099963" watchObservedRunningTime="2026-10-30 04:55:13.348007565 +0000 UTC m=+686.317502553" Oct 30 04:55:14.159270 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:14.159232 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:15.150975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:15.150936 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:15.163657 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:15.163631 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:15.326193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:15.326157 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" Oct 30 04:55:19.193899 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:55:19.193860 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:55:20.331664 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:20.331630 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:20.344467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:20.344446 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:20.355689 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:20.355668 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh" Oct 30 04:55:22.159691 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:22.159656 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:22.171994 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:22.171968 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:22.364657 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:22.364626 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-snw9d" Oct 30 04:55:25.802784 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.802750 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 04:55:25.809735 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.809708 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.825687 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.825661 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dr8cs\"" Oct 30 04:55:25.841365 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.841333 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 04:55:25.879767 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.879721 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.879970 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.879775 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.879970 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.879902 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.879969 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.879995 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.880032 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.880074 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880277 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.880102 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.880277 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.880138 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgkp\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981140 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981102 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgkp\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981154 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981193 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981270 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981307 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981333 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981353 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981715 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981388 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981715 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981425 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981866 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981841 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981933 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981874 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.981933 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.981900 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.982065 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.982043 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.982193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.982175 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.983752 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.983731 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:25.983898 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:25.983880 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:26.001715 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:26.001689 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:26.001860 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:26.001825 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgkp\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:26.123877 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:26.123792 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:26.438483 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:26.438452 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 04:55:26.438727 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:55:26.438703 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230217f_bdf0_48a5_a885_14836a50bc22.slice/crio-6427010f444aa6d703fa7a13baa08a4d3690b009b6fd3002eb88ae4d85a3841e WatchSource:0}: Error finding container 6427010f444aa6d703fa7a13baa08a4d3690b009b6fd3002eb88ae4d85a3841e: Status 404 returned error can't find the container with id 6427010f444aa6d703fa7a13baa08a4d3690b009b6fd3002eb88ae4d85a3841e Oct 30 04:55:27.377741 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:27.377697 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" event={"ID":"2230217f-bdf0-48a5-a885-14836a50bc22","Type":"ContainerStarted","Data":"6427010f444aa6d703fa7a13baa08a4d3690b009b6fd3002eb88ae4d85a3841e"} Oct 30 04:55:29.007842 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:29.007811 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:29.008123 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:29.007878 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:29.008123 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:29.007908 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:29.391004 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:29.390972 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" event={"ID":"2230217f-bdf0-48a5-a885-14836a50bc22","Type":"ContainerStarted","Data":"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516"} Oct 30 04:55:29.460905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:29.460857 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podStartSLOduration=1.894516387 podStartE2EDuration="4.460845513s" totalImagesPullingTime="2.566329126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:29.447525566 +0000 UTC m=+702.417020568" watchObservedRunningTime="2026-10-30 04:55:29.460845513 +0000 UTC m=+702.430340503" Oct 30 04:55:30.148239 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:30.133625 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:30.149427 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:30.149387 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:30.149582 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:30.149473 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:31.124344 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:31.124303 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:31.124674 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:31.124640 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:32.138659 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:32.138617 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:32.139263 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:32.139206 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:33.130653 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:33.130614 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:33.130933 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:33.130902 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:34.131242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:34.130592 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:34.131242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:34.130672 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:35.124580 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:35.124542 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:35.124758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:35.124614 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:36.124763 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:36.124731 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 04:55:36.125156 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:36.124780 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 04:55:38.125332 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:38.125287 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:55:38.125764 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:38.125359 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:55:39.166435 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:39.166389 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:55:39.166914 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:39.166460 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:55:39.374698 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:39.374663 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:39.375404 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:39.375375 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:39.376609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:39.376588 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 04:55:40.861259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.861206 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 04:55:40.864789 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.864772 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.898069 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.898032 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 04:55:40.952431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952403 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952448 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952466 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952482 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952593 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952826 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952623 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952826 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952668 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952826 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952777 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:40.952958 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:40.952833 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdgb\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.053783 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.053744 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdgb\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.053961 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.053791 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.053961 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.053921 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.053961 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.053955 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.053971 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054008 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054024 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054122 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054089 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054374 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054134 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054434 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054387 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054434 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054409 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054624 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054606 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054696 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054670 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.054821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.054795 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.056115 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.056091 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.056420 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.056402 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.074169 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.074140 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.074297 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.074206 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdgb\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-672ql\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.178392 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.178363 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:41.482736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.482706 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 04:55:41.484291 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:55:41.484266 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31475025_84fc_4d3f_acc3_2241c6eaa032.slice/crio-72ed093fb9e6d93d2cf1c62e79a80de2eadcdd3a47b315834e69dc38e8c77628 WatchSource:0}: Error finding container 72ed093fb9e6d93d2cf1c62e79a80de2eadcdd3a47b315834e69dc38e8c77628: Status 404 returned error can't find the container with id 72ed093fb9e6d93d2cf1c62e79a80de2eadcdd3a47b315834e69dc38e8c77628 Oct 30 04:55:41.487128 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.487097 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:41.487204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.487166 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:41.487204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:41.487196 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:42.455659 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:42.455622 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" event={"ID":"31475025-84fc-4d3f-acc3-2241c6eaa032","Type":"ContainerStarted","Data":"165e149deb2f6722036de332086b502cbee9145da7d541b1dcbcb7c761e34a9a"} Oct 30 04:55:42.456057 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:42.455676 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" event={"ID":"31475025-84fc-4d3f-acc3-2241c6eaa032","Type":"ContainerStarted","Data":"72ed093fb9e6d93d2cf1c62e79a80de2eadcdd3a47b315834e69dc38e8c77628"} Oct 30 04:55:42.514601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:42.514549 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podStartSLOduration=2.514536591 podStartE2EDuration="2.514536591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:42.512099873 +0000 UTC m=+715.481594861" watchObservedRunningTime="2026-10-30 04:55:42.514536591 +0000 UTC m=+715.484031579" Oct 30 04:55:43.178884 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:43.178853 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:43.180291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:43.180269 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:43.180344 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:43.180328 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:44.178997 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:44.178963 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:44.179451 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:44.179015 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:45.179077 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:45.179043 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:45.179494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:45.179093 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:46.178849 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:46.178808 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:46.179064 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:46.178872 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:47.179580 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:47.179549 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:47.180042 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:47.179616 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:48.179461 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:48.179421 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 04:55:48.179670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:48.179480 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 04:55:49.190957 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:55:49.190917 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:55:49.737945 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:49.737896 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:49.738285 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:49.738259 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:55:49.739130 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:55:49.739110 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 04:56:19.189326 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:19.189287 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:56:35.870320 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.870284 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 04:56:35.874072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.874052 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.892262 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.890193 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-397179-openshift-default-dockercfg-q4dsf\"" Oct 30 04:56:35.906498 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.906465 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 04:56:35.964281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964236 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964485 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964356 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9vc\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964485 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964397 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964485 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964429 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964499 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964603 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964624 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964683 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964648 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:35.964806 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:35.964728 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.065791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065753 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.065791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065792 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065817 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065869 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065945 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.065988 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9vc\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066023 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066017 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066048 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066106 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066210 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066419 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066318 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066634 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.066789 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.066678 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.067036 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.067013 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.068481 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.068455 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.068756 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.068733 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.089097 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.089071 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.089299 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.089102 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9vc\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.137001 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.136973 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 04:56:36.142492 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.142465 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.163197 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.163163 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 04:56:36.207642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.207609 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:36.212689 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.212657 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b21507-openshift-default-dockercfg-q4ptg\"" Oct 30 04:56:36.268736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.268694 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.268878 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.268753 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.268878 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.268806 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.268878 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.268832 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.269049 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.268970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.269118 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.269100 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.269172 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.269153 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.273267 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.269210 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.273267 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.269286 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97qr\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.274796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.274769 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 04:56:36.283239 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.283118 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.301668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.301354 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 04:56:36.301668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.301561 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-889ca9-openshift-default-dockercfg-t56wr\"" Oct 30 04:56:36.369814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369780 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.369814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369817 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370037 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369849 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370037 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369934 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370037 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369957 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370037 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.369984 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370037 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370024 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97qr\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370046 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370095 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370355 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370261 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370355 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370318 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtxw\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370458 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370382 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370458 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370414 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370553 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370468 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370553 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370476 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370554 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370624 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370632 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370803 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370653 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.370803 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370717 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370803 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370736 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.370978 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.370949 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.371111 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.371090 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.372703 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.372683 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.372912 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.372895 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.392107 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.392032 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.395736 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.395708 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97qr\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.457485 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.457448 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:36.472188 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472163 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472207 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472242 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472283 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472306 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472349 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472391 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtxw\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472454 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472686 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472494 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472686 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472630 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.472853 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472834 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.473001 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472975 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.473001 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.472994 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.473353 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.473334 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.475044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.474974 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.475044 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.474983 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.494026 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.493989 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.494170 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.494041 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtxw\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.522619 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.522554 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 04:56:36.533019 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.532976 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.533150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.533064 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.533150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.533108 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.534188 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.534167 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 04:56:36.549846 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.549822 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.559074 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.559039 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 04:56:36.566658 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.566624 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0b21ba-openshift-default-dockercfg-5n9kc\"" Oct 30 04:56:36.606117 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.606092 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:36.694260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.685974 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.686025 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.686054 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.686146 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.686184 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694706 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.686212 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694836 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.694761 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694836 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.694820 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdxh\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.694940 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.694876 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.725080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.724987 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerStarted","Data":"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42"} Oct 30 04:56:36.725080 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.725062 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerStarted","Data":"a72a870819bf8ae98d101c5609d85ecf054b51e58e6e694115d55780497311bd"} Oct 30 04:56:36.762937 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.762903 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 04:56:36.763213 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:36.763191 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451878c2_d0e7_4e41_8cab_80d0bd8f57d1.slice/crio-f2cf04991024b38609587a3edfb5f3616357edf50195f8e96acc1319ee9d5166 WatchSource:0}: Error finding container f2cf04991024b38609587a3edfb5f3616357edf50195f8e96acc1319ee9d5166: Status 404 returned error can't find the container with id f2cf04991024b38609587a3edfb5f3616357edf50195f8e96acc1319ee9d5166 Oct 30 04:56:36.766795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.766755 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.766906 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.766837 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.766906 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.766874 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.767592 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.767554 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podStartSLOduration=1.767542068 podStartE2EDuration="1.767542068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:36.766724072 +0000 UTC m=+769.736219075" watchObservedRunningTime="2026-10-30 04:56:36.767542068 +0000 UTC m=+769.737037056" Oct 30 04:56:36.796495 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796465 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796644 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796569 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796644 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796608 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796644 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796638 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796666 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796711 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdxh\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796749 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.796812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796801 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.797008 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.796832 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.797597 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.797575 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.798469 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.797978 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.798615 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.798048 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.798748 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.798180 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.798748 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.798390 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.800104 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.800083 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.800809 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.800791 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.817740 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.817715 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdxh\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.818099 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.818079 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.872561 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.872515 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:36.915409 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.915378 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 04:56:36.917829 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:36.917754 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c62b671_8729_4713_b94c_1c02da5e1388.slice/crio-df5d09b7f5a1b94614e50d2e4efa2b1d4b7ec3ff161086e8d0632b79d51ed531 WatchSource:0}: Error finding container df5d09b7f5a1b94614e50d2e4efa2b1d4b7ec3ff161086e8d0632b79d51ed531: Status 404 returned error can't find the container with id df5d09b7f5a1b94614e50d2e4efa2b1d4b7ec3ff161086e8d0632b79d51ed531 Oct 30 04:56:36.921192 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.921159 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.921318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.921259 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.921318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:36.921309 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.181255 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:37.180645 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b40eef_545d_4cc5_8ca3_3de7ff375188.slice/crio-dbe8c28a40b20f13e593162f4a854b3c8975f3dd26926806587b5ff1d6ddb875 WatchSource:0}: Error finding container dbe8c28a40b20f13e593162f4a854b3c8975f3dd26926806587b5ff1d6ddb875: Status 404 returned error can't find the container with id dbe8c28a40b20f13e593162f4a854b3c8975f3dd26926806587b5ff1d6ddb875 Oct 30 04:56:37.182017 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.181991 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 04:56:37.183132 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.183099 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.183276 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.183160 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.183276 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.183188 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.208213 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.208187 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:37.214195 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.214172 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:37.732069 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.732032 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerStarted","Data":"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18"} Oct 30 04:56:37.732289 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.732100 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerStarted","Data":"dbe8c28a40b20f13e593162f4a854b3c8975f3dd26926806587b5ff1d6ddb875"} Oct 30 04:56:37.733951 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.733922 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerStarted","Data":"b1e6adbcfc8492f233c58d1241871bd062d656c42c268a27f5230867d1a83dab"} Oct 30 04:56:37.734081 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.733971 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerStarted","Data":"df5d09b7f5a1b94614e50d2e4efa2b1d4b7ec3ff161086e8d0632b79d51ed531"} Oct 30 04:56:37.735751 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.735725 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerStarted","Data":"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704"} Oct 30 04:56:37.735854 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.735771 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerStarted","Data":"f2cf04991024b38609587a3edfb5f3616357edf50195f8e96acc1319ee9d5166"} Oct 30 04:56:37.735991 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.735973 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:37.736956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.736934 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:56:37.776144 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.776089 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podStartSLOduration=1.7760758349999999 podStartE2EDuration="1.776075835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.774534343 +0000 UTC m=+770.744029344" watchObservedRunningTime="2026-10-30 04:56:37.776075835 +0000 UTC m=+770.745570871" Oct 30 04:56:37.817757 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.817601 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podStartSLOduration=1.817584062 podStartE2EDuration="1.817584062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.81565116 +0000 UTC m=+770.785146149" watchObservedRunningTime="2026-10-30 04:56:37.817584062 +0000 UTC m=+770.787079052" Oct 30 04:56:37.859014 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.858963 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podStartSLOduration=1.858949415 podStartE2EDuration="1.858949415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.857584991 +0000 UTC m=+770.827079980" watchObservedRunningTime="2026-10-30 04:56:37.858949415 +0000 UTC m=+770.828444404" Oct 30 04:56:37.872661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:37.872630 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:38.458540 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.458503 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:38.463582 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.463559 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:38.607198 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.607159 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:38.612560 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.612532 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:38.741259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.741136 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:38.742161 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.741486 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:38.742161 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.741731 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:56:38.742161 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.741882 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:56:38.883392 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:38.883359 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:39.745631 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:39.745601 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:39.746737 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:39.746716 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:56:42.819018 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.818980 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 04:56:42.824106 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.824082 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:42.841102 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.841074 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-397179-serving-cert\"" Oct 30 04:56:42.877112 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.877078 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 04:56:42.978287 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.978253 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:42.978444 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.978296 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:42.978488 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.978451 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:42.978532 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:42.978494 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvqj\" (UniqueName: \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.080006 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.079925 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvqj\" (UniqueName: \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.080150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.080001 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.080150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.080029 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.080150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.080129 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.080825 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.080805 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.082656 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.082631 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.082803 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.082781 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.100819 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.100793 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvqj\" (UniqueName: \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\") pod \"maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.138008 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.137978 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:43.481725 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.481695 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 04:56:43.481838 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:43.481813 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c71880_4ac8_4f66_91e3_66f38ca38ea7.slice/crio-c08d4b9652e3b3380e7f51dc2ad123bb1e8cbdb1edde67265b397c5f91411ec6 WatchSource:0}: Error finding container c08d4b9652e3b3380e7f51dc2ad123bb1e8cbdb1edde67265b397c5f91411ec6: Status 404 returned error can't find the container with id c08d4b9652e3b3380e7f51dc2ad123bb1e8cbdb1edde67265b397c5f91411ec6 Oct 30 04:56:43.768857 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:43.768774 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" event={"ID":"32c71880-4ac8-4f66-91e3-66f38ca38ea7","Type":"ContainerStarted","Data":"c08d4b9652e3b3380e7f51dc2ad123bb1e8cbdb1edde67265b397c5f91411ec6"} Oct 30 04:56:44.775073 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.774996 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" event={"ID":"32c71880-4ac8-4f66-91e3-66f38ca38ea7","Type":"ContainerStarted","Data":"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d"} Oct 30 04:56:44.775559 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.775259 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:44.819722 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.819658 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" podStartSLOduration=2.441009368 podStartE2EDuration="2.819641245s" totalImagesPullingTime="378.631877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:44.817725811 +0000 UTC m=+777.787220790" watchObservedRunningTime="2026-10-30 04:56:44.819641245 +0000 UTC m=+777.789136234" Oct 30 04:56:44.845571 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.845535 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 04:56:44.849304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.849283 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.864987 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.864957 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 04:56:44.867341 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.867312 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-cff4ac-serving-cert\"" Oct 30 04:56:44.896418 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.896375 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.896601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.896430 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.896601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.896492 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.896601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.896527 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.997702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.997669 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.997897 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.997713 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.997897 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.997801 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.997897 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.997830 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:44.998066 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:44.997926 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-serving-cert: secret "maas-api-e2e-worker-w1-cff4ac-serving-cert" not found Oct 30 04:56:44.998066 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:44.998002 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls podName:dc3b3b52-4661-4930-a6b1-08e0c350337b nodeName:}" failed. No retries permitted until 2026-10-30 04:56:45.49797935 +0000 UTC m=+778.467474336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls") pod "maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" (UID: "dc3b3b52-4661-4930-a6b1-08e0c350337b") : secret "maas-api-e2e-worker-w1-cff4ac-serving-cert" not found Oct 30 04:56:44.998501 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:44.998482 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:45.000178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:45.000157 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:45.018442 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:45.018415 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:45.504058 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:45.504027 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:45.506429 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:45.506407 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:45.763475 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:45.763114 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:46.092262 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.089281 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 04:56:46.096896 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:46.096782 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3b3b52_4661_4930_a6b1_08e0c350337b.slice/crio-ddb0db5c497b9f28084ce759ae64eb367931087f2453f759f20215940cd90fd4 WatchSource:0}: Error finding container ddb0db5c497b9f28084ce759ae64eb367931087f2453f759f20215940cd90fd4: Status 404 returned error can't find the container with id ddb0db5c497b9f28084ce759ae64eb367931087f2453f759f20215940cd90fd4 Oct 30 04:56:46.442720 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.442693 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 04:56:46.447269 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.447248 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.465864 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.465581 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 04:56:46.468139 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.468112 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0b21ba\"" Oct 30 04:56:46.511780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.511736 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.511948 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.511920 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tnk\" (UniqueName: \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.613078 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.613042 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.613262 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.613133 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tnk\" (UniqueName: \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.613796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.613776 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.620977 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.620924 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 04:56:46.625368 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.625345 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.636124 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.636096 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tnk\" (UniqueName: \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\") pod \"payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.644307 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.644273 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 04:56:46.644837 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.644802 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-0b21ba-serving-cert\"" Oct 30 04:56:46.722164 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.722062 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.722164 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.722147 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfvm\" (UniqueName: \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.722431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.722202 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.722431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.722256 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.767047 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.767014 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:46.802072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.802035 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" event={"ID":"dc3b3b52-4661-4930-a6b1-08e0c350337b","Type":"ContainerStarted","Data":"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31"} Oct 30 04:56:46.802449 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.802099 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" event={"ID":"dc3b3b52-4661-4930-a6b1-08e0c350337b","Type":"ContainerStarted","Data":"ddb0db5c497b9f28084ce759ae64eb367931087f2453f759f20215940cd90fd4"} Oct 30 04:56:46.802987 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.802960 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:46.827642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.827164 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.827642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.827276 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfvm\" (UniqueName: \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.827642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.827343 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.827642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.827369 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.828833 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.828441 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.828833 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:46.828637 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-serving-cert: secret "maas-api-e2e-worker-w5-0b21ba-serving-cert" not found Oct 30 04:56:46.828833 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:46.828689 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls podName:aeb17cf5-c745-487d-b7cc-747255962479 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:47.328674491 +0000 UTC m=+780.298169459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls") pod "maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" (UID: "aeb17cf5-c745-487d-b7cc-747255962479") : secret "maas-api-e2e-worker-w5-0b21ba-serving-cert" not found Oct 30 04:56:46.842016 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.841982 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:46.852884 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:46.852855 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfvm\" (UniqueName: \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:47.063888 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.063838 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" podStartSLOduration=2.719565921 podStartE2EDuration="3.063824493s" totalImagesPullingTime="344.258572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:46.847665919 +0000 UTC m=+779.817160908" watchObservedRunningTime="2026-10-30 04:56:47.063824493 +0000 UTC m=+780.033319481" Oct 30 04:56:47.065783 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.065754 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 04:56:47.066435 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:47.066405 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9c17ec_b328_4605_8f42_b85013ce2798.slice/crio-43c64dae2dad37485e644657eca41c2c922084c032db0d9d9061132eacc21156 WatchSource:0}: Error finding container 43c64dae2dad37485e644657eca41c2c922084c032db0d9d9061132eacc21156: Status 404 returned error can't find the container with id 43c64dae2dad37485e644657eca41c2c922084c032db0d9d9061132eacc21156 Oct 30 04:56:47.334160 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.334075 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:47.336560 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.336536 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:47.539018 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.538981 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:47.809820 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.809778 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" event={"ID":"7a9c17ec-b328-4605-8f42-b85013ce2798","Type":"ContainerStarted","Data":"43c64dae2dad37485e644657eca41c2c922084c032db0d9d9061132eacc21156"} Oct 30 04:56:47.855252 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:47.849983 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 04:56:47.858438 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:47.858402 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb17cf5_c745_487d_b7cc_747255962479.slice/crio-545bde26b36fe33e54fc76e4b7bf536da9c7b153dc6a98df738fa041eb00efa0 WatchSource:0}: Error finding container 545bde26b36fe33e54fc76e4b7bf536da9c7b153dc6a98df738fa041eb00efa0: Status 404 returned error can't find the container with id 545bde26b36fe33e54fc76e4b7bf536da9c7b153dc6a98df738fa041eb00efa0 Oct 30 04:56:48.819047 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:48.818145 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" event={"ID":"aeb17cf5-c745-487d-b7cc-747255962479","Type":"ContainerStarted","Data":"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819"} Oct 30 04:56:48.819047 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:48.818265 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" event={"ID":"aeb17cf5-c745-487d-b7cc-747255962479","Type":"ContainerStarted","Data":"545bde26b36fe33e54fc76e4b7bf536da9c7b153dc6a98df738fa041eb00efa0"} Oct 30 04:56:48.819047 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:48.818495 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:48.877180 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:48.877114 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" podStartSLOduration=2.548996207 podStartE2EDuration="2.877092717s" totalImagesPullingTime="328.09651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:48.874760428 +0000 UTC m=+781.844255418" watchObservedRunningTime="2026-10-30 04:56:48.877092717 +0000 UTC m=+781.846587706" Oct 30 04:56:49.193019 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:49.192963 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:56:50.424163 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.424123 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 04:56:50.463521 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.463491 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 04:56:50.463712 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.463621 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.479199 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.479171 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b21507-serving-cert\"" Oct 30 04:56:50.499634 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.499601 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmkq\" (UniqueName: \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.499776 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.499650 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.499776 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.499718 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.499907 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.499875 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.601343 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.601305 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.601515 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.601377 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmkq\" (UniqueName: \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.601515 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.601423 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.601515 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.601443 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.601963 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:50.601941 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-serving-cert: secret "maas-api-e2e-worker-w0-b21507-serving-cert" not found Oct 30 04:56:50.602051 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:50.602038 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls podName:a03b79b2-9a5a-47cf-b430-8562ca63ceef nodeName:}" failed. No retries permitted until 2026-10-30 04:56:51.102006333 +0000 UTC m=+784.071501317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls") pod "maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" (UID: "a03b79b2-9a5a-47cf-b430-8562ca63ceef") : secret "maas-api-e2e-worker-w0-b21507-serving-cert" not found Oct 30 04:56:50.602317 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.602146 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.604004 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.603986 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.663290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.662943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmkq\" (UniqueName: \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:50.798283 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.798175 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 04:56:50.855242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.855160 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" event={"ID":"7a9c17ec-b328-4605-8f42-b85013ce2798","Type":"ContainerStarted","Data":"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e"} Oct 30 04:56:50.855434 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.855279 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:56:50.896932 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:50.896883 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" podStartSLOduration=1.906649716 podStartE2EDuration="4.896871313s" totalImagesPullingTime="2.990221597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:50.896499013 +0000 UTC m=+783.865994019" watchObservedRunningTime="2026-10-30 04:56:50.896871313 +0000 UTC m=+783.866366305" Oct 30 04:56:51.109738 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:51.109662 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:51.112780 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:51.112761 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:51.378765 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:51.378684 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:51.706513 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:51.704427 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 04:56:51.711014 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:51.710983 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03b79b2_9a5a_47cf_b430_8562ca63ceef.slice/crio-3ca25f488565e7a3bf8b7b3515327e288a44def1772a0e4133fa6e94e1c284ba WatchSource:0}: Error finding container 3ca25f488565e7a3bf8b7b3515327e288a44def1772a0e4133fa6e94e1c284ba: Status 404 returned error can't find the container with id 3ca25f488565e7a3bf8b7b3515327e288a44def1772a0e4133fa6e94e1c284ba Oct 30 04:56:51.862578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:51.862536 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" event={"ID":"a03b79b2-9a5a-47cf-b430-8562ca63ceef","Type":"ContainerStarted","Data":"3ca25f488565e7a3bf8b7b3515327e288a44def1772a0e4133fa6e94e1c284ba"} Oct 30 04:56:52.413280 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.413241 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 04:56:52.417302 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.417279 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.429465 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.425815 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.429465 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.425873 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84w5\" (UniqueName: \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.429465 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.426015 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.429465 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.426091 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.456242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.454145 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d55037-serving-cert\"" Oct 30 04:56:52.470441 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.469383 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.533864 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.533956 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.533996 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.534034 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84w5\" (UniqueName: \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:52.534490 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-serving-cert: secret "maas-api-e2e-worker-w4-d55037-serving-cert" not found Oct 30 04:56:52.544290 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:52.534557 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls podName:3bf3ed89-c26a-43c2-8ec7-7db1888117d4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:53.034537019 +0000 UTC m=+786.004031989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls") pod "maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" (UID: "3bf3ed89-c26a-43c2-8ec7-7db1888117d4") : secret "maas-api-e2e-worker-w4-d55037-serving-cert" not found Oct 30 04:56:52.548241 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.547088 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.588918 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.588827 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.589079 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.589038 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84w5\" (UniqueName: \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:52.881976 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.881872 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" event={"ID":"a03b79b2-9a5a-47cf-b430-8562ca63ceef","Type":"ContainerStarted","Data":"5d0b9ec3a9f512f84ec78a149d43d3de70c5f91de5a1f064479e8266a58e786e"} Oct 30 04:56:52.882484 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.882320 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:52.947240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:52.946859 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" podStartSLOduration=2.609618962 podStartE2EDuration="2.946824922s" totalImagesPullingTime="337.20596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:52.945413203 +0000 UTC m=+785.914908191" watchObservedRunningTime="2026-10-30 04:56:52.946824922 +0000 UTC m=+785.916319912" Oct 30 04:56:53.072238 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.069828 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:53.093243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.091559 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:53.377604 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.377569 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:53.688717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.688144 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 04:56:53.688717 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:53.688650 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf3ed89_c26a_43c2_8ec7_7db1888117d4.slice/crio-92971e008d5c28ac0a85c37a933f1943427d424f769e7371926e10f6c65c81e3 WatchSource:0}: Error finding container 92971e008d5c28ac0a85c37a933f1943427d424f769e7371926e10f6c65c81e3: Status 404 returned error can't find the container with id 92971e008d5c28ac0a85c37a933f1943427d424f769e7371926e10f6c65c81e3 Oct 30 04:56:53.847242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.838395 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 04:56:53.892261 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:53.891160 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" event={"ID":"3bf3ed89-c26a-43c2-8ec7-7db1888117d4","Type":"ContainerStarted","Data":"92971e008d5c28ac0a85c37a933f1943427d424f769e7371926e10f6c65c81e3"} Oct 30 04:56:54.249154 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.249112 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 04:56:54.256473 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.256443 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.260252 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.259353 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 04:56:54.271895 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.271874 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-83e83e-serving-cert\"" Oct 30 04:56:54.286186 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.286153 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.286477 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.286457 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.286701 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.286679 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.286821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.286786 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.388039 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.388010 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.388204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.388053 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.388204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.388135 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.388204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.388171 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.388422 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:54.388255 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-serving-cert: secret "maas-api-e2e-worker-w6-83e83e-serving-cert" not found Oct 30 04:56:54.388476 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:56:54.388342 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls podName:f6e2856c-2432-472d-9930-2a7ef63e75ab nodeName:}" failed. No retries permitted until 2026-10-30 04:56:54.888320904 +0000 UTC m=+787.857815885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls") pod "maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" (UID: "f6e2856c-2432-472d-9930-2a7ef63e75ab") : secret "maas-api-e2e-worker-w6-83e83e-serving-cert" not found Oct 30 04:56:54.389278 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.389250 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.390860 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.390831 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.408911 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.408874 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.854388 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.854311 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 04:56:54.893260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.893030 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.902266 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.897918 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") pod \"maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:54.902266 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.901631 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" event={"ID":"3bf3ed89-c26a-43c2-8ec7-7db1888117d4","Type":"ContainerStarted","Data":"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9"} Oct 30 04:56:54.902266 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.901771 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:56:54.958448 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:54.958377 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" podStartSLOduration=2.590190997 podStartE2EDuration="2.958360154s" totalImagesPullingTime="368.169157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:54.95779691 +0000 UTC m=+787.927291970" watchObservedRunningTime="2026-10-30 04:56:54.958360154 +0000 UTC m=+787.927855143" Oct 30 04:56:55.175617 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:55.175583 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:55.503193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:55.503161 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 04:56:55.504906 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:56:55.504868 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e2856c_2432_472d_9930_2a7ef63e75ab.slice/crio-79c80fd2f590c993266f80ef36a65b79de020395057d040a7dd032b648a922ca WatchSource:0}: Error finding container 79c80fd2f590c993266f80ef36a65b79de020395057d040a7dd032b648a922ca: Status 404 returned error can't find the container with id 79c80fd2f590c993266f80ef36a65b79de020395057d040a7dd032b648a922ca Oct 30 04:56:55.910758 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:55.910713 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" event={"ID":"f6e2856c-2432-472d-9930-2a7ef63e75ab","Type":"ContainerStarted","Data":"79c80fd2f590c993266f80ef36a65b79de020395057d040a7dd032b648a922ca"} Oct 30 04:56:56.917578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:56.917541 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" event={"ID":"f6e2856c-2432-472d-9930-2a7ef63e75ab","Type":"ContainerStarted","Data":"e2c3aa3b48bc3d0527754c8d1fec84f25a475355267ff6f07ce4848aef28ffe6"} Oct 30 04:56:56.918096 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:56.917674 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:56:56.960867 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:56.960816 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" podStartSLOduration=2.590078788 podStartE2EDuration="2.960803792s" totalImagesPullingTime="370.725004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:56.96042206 +0000 UTC m=+789.929917073" watchObservedRunningTime="2026-10-30 04:56:56.960803792 +0000 UTC m=+789.930298781" Oct 30 04:56:58.898315 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.898289 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 04:56:58.956964 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.956095 5058 generic.go:356] "Generic (PLEG): container finished" podID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerID="82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42" exitCode=0 Oct 30 04:56:58.956964 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.956295 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerDied","Data":"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42"} Oct 30 04:56:58.956964 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.956753 5058 scope.go:122] "RemoveContainer" containerID="82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42" Oct 30 04:56:58.961389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.959632 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:58.961389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.959704 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:58.961389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:58.959762 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:59.978743 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:56:59.978697 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerStarted","Data":"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1"} Oct 30 04:57:00.208293 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:00.208256 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:57:00.208497 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:00.208473 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:00.208557 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:00.208533 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:00.917948 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:00.917916 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 04:57:01.208571 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:01.208493 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:01.208571 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:01.208554 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:01.873915 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:01.873881 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 04:57:02.208959 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:02.208919 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:02.209476 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:02.208979 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:02.959244 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:02.958428 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 04:57:03.208867 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:03.208828 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:03.209159 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:03.209129 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:04.210120 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:04.208327 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:04.210120 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:04.208408 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:05.208470 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:05.208427 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 04:57:05.208689 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:05.208513 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 04:57:06.059175 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.059136 5058 generic.go:356] "Generic (PLEG): container finished" podID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerID="4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18" exitCode=0 Oct 30 04:57:06.059706 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.059209 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerDied","Data":"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18"} Oct 30 04:57:06.059706 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.059628 5058 scope.go:122] "RemoveContainer" containerID="4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18" Oct 30 04:57:06.060928 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.060896 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:06.061033 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.060965 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:06.061033 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.060998 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:06.213303 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.213275 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:57:06.213661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.213639 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:57:06.214599 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:06.214580 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 04:57:07.081944 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:07.081910 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerStarted","Data":"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0"} Oct 30 04:57:07.874383 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:07.874289 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:57:07.875038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:07.874684 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 04:57:07.875038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:07.874738 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 04:57:08.751246 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:08.750348 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:08.751246 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:08.750412 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:08.873995 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:08.873903 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 04:57:08.873995 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:08.873970 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 04:57:09.096892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.096799 5058 generic.go:356] "Generic (PLEG): container finished" podID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerID="90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704" exitCode=0 Oct 30 04:57:09.096892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.096866 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerDied","Data":"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704"} Oct 30 04:57:09.097290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.097275 5058 scope.go:122] "RemoveContainer" containerID="90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704" Oct 30 04:57:09.098816 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.098784 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:09.098892 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.098874 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:09.098939 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.098929 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:09.750243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.749338 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:57:09.791390 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.791347 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:57:09.791796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.791420 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:57:09.880243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.876322 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 04:57:09.880243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:09.876386 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 04:57:10.114389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.114315 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerStarted","Data":"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771"} Oct 30 04:57:10.462082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.461634 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:57:10.462082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.461763 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:10.462082 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.461804 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:10.874165 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.873646 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 04:57:10.874165 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:10.873707 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 04:57:11.459131 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:11.458657 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:11.459131 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:11.458721 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:11.874284 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:11.873176 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 04:57:11.874284 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:11.873256 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 04:57:12.458653 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:12.458609 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:12.458937 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:12.458905 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:12.878971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:12.878900 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:57:12.880618 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:12.880594 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 04:57:13.458282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:13.458241 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:13.458470 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:13.458336 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:14.181435 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.181400 5058 generic.go:356] "Generic (PLEG): container finished" podID="9c62b671-8729-4713-b94c-1c02da5e1388" containerID="b1e6adbcfc8492f233c58d1241871bd062d656c42c268a27f5230867d1a83dab" exitCode=0 Oct 30 04:57:14.182001 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.181468 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerDied","Data":"b1e6adbcfc8492f233c58d1241871bd062d656c42c268a27f5230867d1a83dab"} Oct 30 04:57:14.182001 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.181926 5058 scope.go:122] "RemoveContainer" containerID="b1e6adbcfc8492f233c58d1241871bd062d656c42c268a27f5230867d1a83dab" Oct 30 04:57:14.183729 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.183688 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:14.183831 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.183786 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:14.183872 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.183837 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:14.461930 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.461599 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 04:57:14.461930 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:14.461663 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 04:57:15.189133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.189073 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerStarted","Data":"c46114eb6d71ab11131c96826ba6763a79e881095092554e3a5c28eef79676e1"} Oct 30 04:57:15.573301 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.573194 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:57:15.573494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.573463 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:57:15.574346 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.574326 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 04:57:15.606885 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.606856 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:57:15.607169 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.607148 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:57:15.607248 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:15.607192 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 04:57:16.606607 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:16.606561 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:57:16.607015 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:16.606636 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 04:57:17.607111 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:17.607073 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:57:17.607529 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:17.607161 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 04:57:18.607165 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:18.607126 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:57:18.607600 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:18.607193 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 04:57:19.191559 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:57:19.191523 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:57:19.607060 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:19.606964 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:57:19.607060 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:19.607019 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 04:57:20.631003 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:20.630956 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:57:20.631813 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:20.631784 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:57:20.633670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:20.633643 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 04:57:49.190162 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:57:49.190124 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:57:51.277326 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.277292 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 04:57:51.282971 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.282942 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.300808 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.300778 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 04:57:51.311600 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.311571 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-83e83e-openshift-default-dockercfg-zgpn5\"" Oct 30 04:57:51.323546 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323512 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.323710 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323664 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.323710 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323701 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xph\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.323866 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323845 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.323924 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323900 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.323975 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.323956 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.324072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.324048 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.324202 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.324095 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.324312 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.324262 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425063 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425027 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425063 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425063 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xph\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425110 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425149 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425196 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425259 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425293 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425339 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425685 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425415 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425685 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425477 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425685 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425519 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425846 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425719 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425905 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425889 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.425949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.425927 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.427760 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.427734 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.427996 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.427976 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.445436 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.445405 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.445566 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.445469 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xph\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.601663 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.601582 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:51.908276 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.908247 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 04:57:51.908979 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:57:51.908952 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082518df_92fd_46d1_9d1b_c8d80dc3d4d4.slice/crio-7a8833444d53841fb7acdc078c99a641448e3574534d4c18cd7b4d64c2f452dc WatchSource:0}: Error finding container 7a8833444d53841fb7acdc078c99a641448e3574534d4c18cd7b4d64c2f452dc: Status 404 returned error can't find the container with id 7a8833444d53841fb7acdc078c99a641448e3574534d4c18cd7b4d64c2f452dc Oct 30 04:57:51.911413 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.911377 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:51.911499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.911441 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:51.911499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:51.911494 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:52.407117 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:52.407082 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" event={"ID":"082518df-92fd-46d1-9d1b-c8d80dc3d4d4","Type":"ContainerStarted","Data":"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8"} Oct 30 04:57:52.407620 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:52.407135 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" event={"ID":"082518df-92fd-46d1-9d1b-c8d80dc3d4d4","Type":"ContainerStarted","Data":"7a8833444d53841fb7acdc078c99a641448e3574534d4c18cd7b4d64c2f452dc"} Oct 30 04:57:52.601840 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:52.601801 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:52.603261 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:52.603229 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:52.603391 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:52.603284 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:53.602839 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:53.602805 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:53.603256 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:53.602854 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:54.601992 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:54.601955 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:54.602178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:54.602017 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:55.602373 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:55.602334 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:55.602796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:55.602392 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:56.602926 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:56.602868 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:56.603370 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:56.602958 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:57.602382 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:57.602337 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:57:57.602553 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:57.602416 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 04:57:58.606389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:58.606358 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:58.606837 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:58.606679 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:58.607421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:58.607404 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 04:57:58.662191 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:57:58.662140 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podStartSLOduration=7.662127082 podStartE2EDuration="7.662127082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:52.456070285 +0000 UTC m=+845.425565275" watchObservedRunningTime="2026-10-30 04:57:58.662127082 +0000 UTC m=+851.631622073" Oct 30 04:58:03.729133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.729093 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 04:58:03.734481 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.734454 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.748178 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.748148 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 04:58:03.759371 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.759337 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"kube-root-ca.crt\"" Oct 30 04:58:03.759517 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.759401 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"default-dockercfg-w4dgw\"" Oct 30 04:58:03.763560 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.763530 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"openshift-service-ca.crt\"" Oct 30 04:58:03.763717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.763578 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"premium-w4-d55037-kserve-self-signed-certs\"" Oct 30 04:58:03.887315 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887280 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266hh\" (UniqueName: \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.887499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887324 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.887499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887447 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.887499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887493 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.887625 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887606 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.887676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.887644 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988703 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988602 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266hh\" (UniqueName: \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988703 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988728 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988774 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988850 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.988949 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.988888 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.989163 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.989149 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.989250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.989148 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.989308 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.989278 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.991193 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.991170 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:03.991517 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:03.991498 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:04.008848 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.008819 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266hh\" (UniqueName: \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\") pod \"premium-w4-d55037-kserve-75bb57ffb6-4m8tm\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:04.057732 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.057697 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:04.354318 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.354288 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 04:58:04.355490 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:58:04.355463 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b1524ea_a722_4060_ad90_ebde189c382b.slice/crio-88e0a0e9f0565c941b813d6c6e1c9c19e9c54bc7725a52fe014f7d1c141b085c WatchSource:0}: Error finding container 88e0a0e9f0565c941b813d6c6e1c9c19e9c54bc7725a52fe014f7d1c141b085c: Status 404 returned error can't find the container with id 88e0a0e9f0565c941b813d6c6e1c9c19e9c54bc7725a52fe014f7d1c141b085c Oct 30 04:58:04.505848 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.505814 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" event={"ID":"7b1524ea-a722-4060-ad90-ebde189c382b","Type":"ContainerStarted","Data":"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042"} Oct 30 04:58:04.505848 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.505852 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" event={"ID":"7b1524ea-a722-4060-ad90-ebde189c382b","Type":"ContainerStarted","Data":"88e0a0e9f0565c941b813d6c6e1c9c19e9c54bc7725a52fe014f7d1c141b085c"} Oct 30 04:58:04.545796 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:04.545746 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podStartSLOduration=1.545729756 podStartE2EDuration="1.545729756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:04.54571187 +0000 UTC m=+857.515206860" watchObservedRunningTime="2026-10-30 04:58:04.545729756 +0000 UTC m=+857.515224746" Oct 30 04:58:05.058291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:05.058253 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:14.058316 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:14.058275 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:14.070708 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:14.070681 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:14.575708 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:14.575677 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 04:58:19.190811 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:58:19.190776 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:58:22.002025 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.001989 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 04:58:22.007542 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.007516 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.026399 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.026367 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 04:58:22.036929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.036899 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d55037-openshift-default-dockercfg-ppf9l\"" Oct 30 04:58:22.105990 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.105957 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106175 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106000 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106175 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106020 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106287 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106183 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106287 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106253 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106305 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106376 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106421 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106396 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnbx\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.106543 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.106431 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.207955 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.207916 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.207985 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208011 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208034 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208088 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208133 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208119 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208454 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208166 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208454 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208272 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208454 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208297 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnbx\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208469 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208524 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208767 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.208791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208776 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.209021 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.208943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.210500 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.210473 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.210814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.210794 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.228791 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.228760 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnbx\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.229233 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.229197 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.321882 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.321800 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:22.650689 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.650654 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 04:58:22.650802 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:58:22.650775 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7049d7_cd5e_4b6b_bd9e_17753d20db18.slice/crio-7a25fc7dc79835cbeb4ef12275d8b6b6e42ec0a645ce40bda734192ab5e7934f WatchSource:0}: Error finding container 7a25fc7dc79835cbeb4ef12275d8b6b6e42ec0a645ce40bda734192ab5e7934f: Status 404 returned error can't find the container with id 7a25fc7dc79835cbeb4ef12275d8b6b6e42ec0a645ce40bda734192ab5e7934f Oct 30 04:58:22.653308 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.653280 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.653396 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.653343 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.653396 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:22.653370 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:23.620401 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:23.620368 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" event={"ID":"de7049d7-cd5e-4b6b-bd9e-17753d20db18","Type":"ContainerStarted","Data":"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7"} Oct 30 04:58:23.620792 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:23.620422 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" event={"ID":"de7049d7-cd5e-4b6b-bd9e-17753d20db18","Type":"ContainerStarted","Data":"7a25fc7dc79835cbeb4ef12275d8b6b6e42ec0a645ce40bda734192ab5e7934f"} Oct 30 04:58:23.691748 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:23.691696 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podStartSLOduration=2.691681587 podStartE2EDuration="2.691681587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:23.677958977 +0000 UTC m=+876.647453965" watchObservedRunningTime="2026-10-30 04:58:23.691681587 +0000 UTC m=+876.661176577" Oct 30 04:58:24.347243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:24.326631 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:24.347243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:24.330367 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 04:58:24.347243 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:24.330420 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 04:58:25.323185 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:25.323144 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 04:58:25.323698 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:25.323244 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 04:58:26.322568 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:26.322527 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 04:58:26.322740 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:26.322590 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 04:58:27.323095 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:27.323058 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 04:58:27.323531 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:27.323115 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 04:58:28.322814 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:28.322776 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 04:58:28.323004 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:28.322833 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 04:58:29.629551 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:29.629514 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:29.631614 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:29.631543 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:29.635530 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:29.635503 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 04:58:47.291706 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:47.291673 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:58:47.294482 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:47.294461 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 04:58:49.190005 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:58:49.189970 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:58:51.284395 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.284358 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 04:58:51.290021 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.289995 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.313586 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.313551 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 04:58:51.355621 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355585 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.355807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355709 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ksz\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.355807 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355753 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.355916 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355819 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.355916 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355886 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.356020 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355927 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.356020 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355954 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.356020 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.355974 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.356163 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.356016 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457501 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457464 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457524 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457552 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457585 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457610 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457642 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457930 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457679 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457930 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457761 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ksz\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.457930 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.457796 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.458104 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.458004 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.458164 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.458102 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.458164 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.458133 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.458256 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.458167 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.458573 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.458555 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.460512 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.460484 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.460811 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.460790 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.485601 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.485575 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.485895 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.485875 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ksz\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.609495 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.609410 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:51.789105 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.789067 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 04:58:51.793604 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.793583 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.811141 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.811115 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 04:58:51.821379 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.821346 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cff4ac-openshift-default-dockercfg-xcpvh\"" Oct 30 04:58:51.862127 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862039 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862127 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862104 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862337 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862190 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862337 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862246 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862337 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862298 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862337 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862319 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmztb\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862482 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862352 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862482 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862466 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.862550 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.862501 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.916786 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:58:51.916755 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7723d7b2_703a_45c2_9a32_a05d048c9d72.slice/crio-2ebcae60a67a5a27a7567db77a8ea1d5369742f96993a385b2bc2964f48ecc4c WatchSource:0}: Error finding container 2ebcae60a67a5a27a7567db77a8ea1d5369742f96993a385b2bc2964f48ecc4c: Status 404 returned error can't find the container with id 2ebcae60a67a5a27a7567db77a8ea1d5369742f96993a385b2bc2964f48ecc4c Oct 30 04:58:51.917197 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.917176 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 04:58:51.919128 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.919099 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:51.919240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.919206 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:51.919309 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.919275 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:51.963114 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963091 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963157 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963282 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963200 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963406 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963353 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963462 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963421 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963516 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963467 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963518 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963578 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963555 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963584 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmztb\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963638 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.963770 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963676 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.964150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963891 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.964150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963962 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.964150 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.963978 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.965797 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.965773 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.966015 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.965998 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.982963 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.982942 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:51.983197 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:51.983180 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmztb\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:52.108114 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.108083 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:52.419103 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:58:52.419070 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccd7794_b6dc_4417_b419_012fdbd4a42f.slice/crio-3d185d213e6f26c12ca7b6db57a16cb8a87502dedd96bcdffe6054c84dcff5ea WatchSource:0}: Error finding container 3d185d213e6f26c12ca7b6db57a16cb8a87502dedd96bcdffe6054c84dcff5ea: Status 404 returned error can't find the container with id 3d185d213e6f26c12ca7b6db57a16cb8a87502dedd96bcdffe6054c84dcff5ea Oct 30 04:58:52.419451 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.419403 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 04:58:52.421450 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.421419 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:52.421522 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.421487 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:52.421522 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.421518 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:52.860612 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.860522 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" event={"ID":"7723d7b2-703a-45c2-9a32-a05d048c9d72","Type":"ContainerStarted","Data":"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2"} Oct 30 04:58:52.860893 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.860867 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" event={"ID":"7723d7b2-703a-45c2-9a32-a05d048c9d72","Type":"ContainerStarted","Data":"2ebcae60a67a5a27a7567db77a8ea1d5369742f96993a385b2bc2964f48ecc4c"} Oct 30 04:58:52.863011 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.862984 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" event={"ID":"9ccd7794-b6dc-4417-b419-012fdbd4a42f","Type":"ContainerStarted","Data":"ec36b2059ef023a510897bb34f01fbf5c905052e4b44fb82ee2ffb5485e16d1d"} Oct 30 04:58:52.863391 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.863036 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" event={"ID":"9ccd7794-b6dc-4417-b419-012fdbd4a42f","Type":"ContainerStarted","Data":"3d185d213e6f26c12ca7b6db57a16cb8a87502dedd96bcdffe6054c84dcff5ea"} Oct 30 04:58:52.903842 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.903790 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podStartSLOduration=1.9037708009999998 podStartE2EDuration="1.903770801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:52.902764108 +0000 UTC m=+905.872259107" watchObservedRunningTime="2026-10-30 04:58:52.903770801 +0000 UTC m=+905.873265790" Oct 30 04:58:52.961891 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:52.961829 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podStartSLOduration=1.961814605 podStartE2EDuration="1.961814605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:52.948561792 +0000 UTC m=+905.918056794" watchObservedRunningTime="2026-10-30 04:58:52.961814605 +0000 UTC m=+905.931309593" Oct 30 04:58:53.108294 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.108254 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:58:53.110120 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.110092 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:53.110270 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.110147 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:53.609765 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.609731 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:58:53.611655 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.611630 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:53.611764 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:53.611692 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:54.109089 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:54.109051 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:54.109301 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:54.109120 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:54.609747 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:54.609709 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:54.609956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:54.609771 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:55.108772 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:55.108722 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:55.108972 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:55.108808 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:55.610405 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:55.610352 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:55.610834 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:55.610421 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:56.109514 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:56.109471 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:56.109707 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:56.109538 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:56.610036 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:56.610004 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:56.610290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:56.610267 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:57.109004 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:57.108972 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:57.109475 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:57.109359 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:57.610338 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:57.610285 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:57.610593 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:57.610364 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:58.110694 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:58.110646 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:58.111373 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:58.111339 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:58:58.618273 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:58.610634 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:58.618273 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:58.610712 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 04:58:59.109728 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:59.109690 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 04:58:59.110030 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:58:59.109999 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 04:59:00.703242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:00.690332 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:59:00.703242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:00.690410 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:01.213035 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.212333 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:59:01.213035 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.212416 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:01.310755 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.310412 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:59:01.313259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.311149 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:59:01.314555 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.314487 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 04:59:01.446851 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.446806 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:59:01.450039 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.449967 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:59:01.482246 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:01.480627 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 04:59:07.006336 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.006301 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 04:59:07.010874 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.010853 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.031118 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.031084 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 04:59:07.059144 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059099 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059313 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059151 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059313 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059256 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxkq\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059313 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059308 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059481 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059436 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059539 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059495 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059598 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059581 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059668 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059643 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.059724 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.059689 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.160908 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxkq\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.160997 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.161026 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.161076 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161660 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.161620 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.161945 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.161622 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.162085 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.161926 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.162085 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.162018 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.162242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.162096 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.162242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.162138 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.162522 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.162501 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.180259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.162765 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.180259 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.170602 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.181890 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.181857 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.193473 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.193440 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.194812 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.194780 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.196289 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.196266 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.198886 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.198867 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxkq\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.330802 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.330714 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:07.648248 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.645947 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 04:59:07.652879 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:07.652838 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dfe251_99ad_4b6f_9b5b_3747c3b334be.slice/crio-294d00a94f431c5d2263522f93f0fc9adc94fd700287cb2e5ea66d15b2df0ffb WatchSource:0}: Error finding container 294d00a94f431c5d2263522f93f0fc9adc94fd700287cb2e5ea66d15b2df0ffb: Status 404 returned error can't find the container with id 294d00a94f431c5d2263522f93f0fc9adc94fd700287cb2e5ea66d15b2df0ffb Oct 30 04:59:07.655458 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.655408 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:07.655579 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.655497 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:07.655640 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:07.655580 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:08.109792 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:08.109697 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" event={"ID":"c2dfe251-99ad-4b6f-9b5b-3747c3b334be","Type":"ContainerStarted","Data":"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c"} Oct 30 04:59:08.109792 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:08.109770 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" event={"ID":"c2dfe251-99ad-4b6f-9b5b-3747c3b334be","Type":"ContainerStarted","Data":"294d00a94f431c5d2263522f93f0fc9adc94fd700287cb2e5ea66d15b2df0ffb"} Oct 30 04:59:08.340244 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:08.338595 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:08.349702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:08.349624 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:08.349702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:08.349689 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:09.338096 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:09.338057 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:09.338844 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:09.338809 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:10.343964 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.343930 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:10.345649 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.345617 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:10.391713 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.391654 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podStartSLOduration=4.39163051 podStartE2EDuration="4.39163051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:08.181508381 +0000 UTC m=+921.151003369" watchObservedRunningTime="2026-10-30 04:59:10.39163051 +0000 UTC m=+923.361125501" Oct 30 04:59:10.394509 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.394477 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 04:59:10.399431 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.399045 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.414251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.414195 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 04:59:10.430251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.424299 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"distinct-w1-cff4ac-kserve-self-signed-certs\"" Oct 30 04:59:10.430251 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.424370 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"default-dockercfg-7dcgl\"" Oct 30 04:59:10.450652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.450618 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"kube-root-ca.crt\"" Oct 30 04:59:10.450973 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.450955 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"openshift-service-ca.crt\"" Oct 30 04:59:10.500188 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500152 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.500441 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500241 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.500441 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500389 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.500558 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500437 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.500609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500572 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.500609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.500602 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fz7\" (UniqueName: \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602353 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602269 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602353 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602313 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fz7\" (UniqueName: \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602353 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602354 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602430 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602495 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602546 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602836 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602777 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602836 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602776 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.602973 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.602944 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.604896 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.604874 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.605156 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.605135 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.683240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.682104 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fz7\" (UniqueName: \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\") pod \"distinct-w1-cff4ac-kserve-64676d85f9-7dpkb\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:10.732587 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:10.732553 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:11.094684 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:11.094651 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 04:59:11.095258 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:11.095230 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod833a27e7_688a_4148_a8ef_71cb6cf67cac.slice/crio-8da8c1392af520302e73a72ecbc7b609ebf09c4cf90870cc6712fd57aee491cf WatchSource:0}: Error finding container 8da8c1392af520302e73a72ecbc7b609ebf09c4cf90870cc6712fd57aee491cf: Status 404 returned error can't find the container with id 8da8c1392af520302e73a72ecbc7b609ebf09c4cf90870cc6712fd57aee491cf Oct 30 04:59:11.163038 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:11.163002 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" event={"ID":"833a27e7-688a-4148-a8ef-71cb6cf67cac","Type":"ContainerStarted","Data":"8da8c1392af520302e73a72ecbc7b609ebf09c4cf90870cc6712fd57aee491cf"} Oct 30 04:59:11.350397 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:11.350347 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:11.350902 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:11.350416 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:12.164897 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:12.164859 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" event={"ID":"833a27e7-688a-4148-a8ef-71cb6cf67cac","Type":"ContainerStarted","Data":"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8"} Oct 30 04:59:12.218580 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:12.218525 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podStartSLOduration=2.218505168 podStartE2EDuration="2.218505168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:12.215737022 +0000 UTC m=+925.185232007" watchObservedRunningTime="2026-10-30 04:59:12.218505168 +0000 UTC m=+925.188000157" Oct 30 04:59:12.334677 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:12.334636 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:12.334992 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:12.334962 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:12.748244 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:12.745316 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:13.345244 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:13.337417 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:13.345244 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:13.337498 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:14.333676 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:14.333637 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 04:59:14.334291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:14.334256 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 04:59:15.613242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:15.610847 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:15.613242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:15.612436 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:15.619250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:15.618514 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 04:59:19.190810 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:59:19.190765 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:59:20.733327 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:20.733287 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:20.746354 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:20.746325 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:21.276670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.276642 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 04:59:21.535693 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.535566 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 04:59:21.540937 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.540913 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.557494 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.557458 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 04:59:21.579324 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579283 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579504 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579420 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579620 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579597 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579695 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579661 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579704 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cqw4\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579742 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579789 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.579929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579825 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.580036 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.579929 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681514 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681469 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681700 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681526 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681700 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681594 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681700 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681692 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681737 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681763 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cqw4\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681794 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681832 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.681879 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681862 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.682137 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.681958 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.682137 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.682080 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.682467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.682270 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.682467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.682390 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.682467 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.682398 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.684019 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.683996 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.684404 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.684371 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.703728 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.703693 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cqw4\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.703728 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.703725 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:21.857499 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:21.857392 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:22.167775 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:22.167746 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3458257c_f753_4c33_8745_496d7f9ff5d9.slice/crio-89cccc068a892ccfa1f856fb5f214859be82708ef2e5b2d3975c98ead45a539d WatchSource:0}: Error finding container 89cccc068a892ccfa1f856fb5f214859be82708ef2e5b2d3975c98ead45a539d: Status 404 returned error can't find the container with id 89cccc068a892ccfa1f856fb5f214859be82708ef2e5b2d3975c98ead45a539d Oct 30 04:59:22.168032 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.168007 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 04:59:22.170432 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.170400 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.170575 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.170463 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.170575 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.170491 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.274474 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.274436 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" event={"ID":"3458257c-f753-4c33-8745-496d7f9ff5d9","Type":"ContainerStarted","Data":"cd245ac0f3b0b18a5cf7d4fd5461060509419a970864ddba5b87803b57b8d558"} Oct 30 04:59:22.274670 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.274503 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" event={"ID":"3458257c-f753-4c33-8745-496d7f9ff5d9","Type":"ContainerStarted","Data":"89cccc068a892ccfa1f856fb5f214859be82708ef2e5b2d3975c98ead45a539d"} Oct 30 04:59:22.326047 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.325995 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podStartSLOduration=1.325981622 podStartE2EDuration="1.325981622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:22.319465316 +0000 UTC m=+935.288960305" watchObservedRunningTime="2026-10-30 04:59:22.325981622 +0000 UTC m=+935.295476611" Oct 30 04:59:22.857932 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:22.857894 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:23.860113 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:23.860078 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:59:23.860547 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:23.860140 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 04:59:24.857864 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:24.857828 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:59:24.858055 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:24.857879 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 04:59:25.858487 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:25.858453 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:59:25.858911 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:25.858505 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 04:59:26.480297 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.480261 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs"] Oct 30 04:59:26.485394 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.485371 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.506134 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.506085 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs"] Oct 30 04:59:26.562242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.562156 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.562440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.562248 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.562440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.562341 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.562440 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.562394 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb29\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-kube-api-access-gfb29\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.562622 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.562460 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.563346 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.563311 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.563468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.563376 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.563468 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.563428 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.563561 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.563502 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.664951 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.664917 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb29\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-kube-api-access-gfb29\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.664986 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665012 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665040 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665078 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665138 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665501 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665232 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665501 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665263 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665501 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665287 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665662 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665602 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665724 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665690 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665724 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665706 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665808 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665776 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.665981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.665963 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.667642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.667615 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.667768 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.667746 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.684860 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.684828 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.685954 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.685936 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb29\" (UniqueName: \"kubernetes.io/projected/3f051c2f-f800-48a2-9a5d-d66a741cdff4-kube-api-access-gfb29\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs\" (UID: \"3f051c2f-f800-48a2-9a5d-d66a741cdff4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.802165 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.802074 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:26.857867 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.857835 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:59:26.858027 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:26.857888 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 04:59:27.151444 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.151406 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs"] Oct 30 04:59:27.155712 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:27.155679 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f051c2f_f800_48a2_9a5d_d66a741cdff4.slice/crio-fab8b41bbc600824bccd265031621f55e2b2350d12947ca2083dfe142bdc92ba WatchSource:0}: Error finding container fab8b41bbc600824bccd265031621f55e2b2350d12947ca2083dfe142bdc92ba: Status 404 returned error can't find the container with id fab8b41bbc600824bccd265031621f55e2b2350d12947ca2083dfe142bdc92ba Oct 30 04:59:27.158386 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.158354 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.158471 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.158419 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.158471 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.158449 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.306723 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.306680 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" event={"ID":"3f051c2f-f800-48a2-9a5d-d66a741cdff4","Type":"ContainerStarted","Data":"c7736fdef1ba3e0ee48309c0ab9e3df4cf959740b2ca1d16459527877f345be7"} Oct 30 04:59:27.306925 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.306744 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" event={"ID":"3f051c2f-f800-48a2-9a5d-d66a741cdff4","Type":"ContainerStarted","Data":"fab8b41bbc600824bccd265031621f55e2b2350d12947ca2083dfe142bdc92ba"} Oct 30 04:59:27.360241 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.352903 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podStartSLOduration=1.352881672 podStartE2EDuration="1.352881672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:27.349687799 +0000 UTC m=+940.319182827" watchObservedRunningTime="2026-10-30 04:59:27.352881672 +0000 UTC m=+940.322376664" Oct 30 04:59:27.802435 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.802350 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:27.858040 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.857999 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:59:27.858212 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:27.858071 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 04:59:28.804779 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:28.804747 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:28.805180 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:28.804800 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:29.277492 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:29.271653 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:29.277492 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:29.272487 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:29.277492 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:29.274295 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 04:59:29.802788 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:29.802751 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:29.802979 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:29.802820 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:30.803194 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:30.803159 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:30.804132 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:30.804096 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:31.802740 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:31.802695 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:31.802934 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:31.802759 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:32.802775 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:32.802737 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:32.803204 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:32.802793 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:33.803322 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:33.803288 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:33.803702 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:33.803345 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:34.236776 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.236739 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 04:59:34.241727 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.241700 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.263384 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.263359 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 04:59:34.266167 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.266135 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"distinct-w3-889ca9-kserve-self-signed-certs\"" Oct 30 04:59:34.266321 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.266278 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"openshift-service-ca.crt\"" Oct 30 04:59:34.266393 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.266379 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"kube-root-ca.crt\"" Oct 30 04:59:34.271797 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.271766 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"default-dockercfg-gf5t6\"" Oct 30 04:59:34.391981 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.391941 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.392157 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.392102 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.392260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.392192 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.392379 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.392342 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.392506 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.392399 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgshr\" (UniqueName: \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.392586 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.392535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496603 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496700 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496735 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496773 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgshr\" (UniqueName: \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496886 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497305 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.496987 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.497756 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.497681 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.501237 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.498584 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.501237 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.498849 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.510823 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.510605 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.513361 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.513334 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.522056 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.522028 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgshr\" (UniqueName: \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\") pod \"distinct-w3-889ca9-kserve-7546d87c76-q4jrj\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.566360 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.566324 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:34.802544 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.802447 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 04:59:34.802544 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.802515 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 04:59:34.870409 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:34.870371 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbc907e0_8871_4947_9711_43ead530bde5.slice/crio-20f08df842f10b03a40a1c97baa5ed7606a1e088c889a8547d40ebda47e33d75 WatchSource:0}: Error finding container 20f08df842f10b03a40a1c97baa5ed7606a1e088c889a8547d40ebda47e33d75: Status 404 returned error can't find the container with id 20f08df842f10b03a40a1c97baa5ed7606a1e088c889a8547d40ebda47e33d75 Oct 30 04:59:34.871168 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:34.871141 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 04:59:35.394994 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:35.394952 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" event={"ID":"fbc907e0-8871-4947-9711-43ead530bde5","Type":"ContainerStarted","Data":"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015"} Oct 30 04:59:35.395198 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:35.395006 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" event={"ID":"fbc907e0-8871-4947-9711-43ead530bde5","Type":"ContainerStarted","Data":"20f08df842f10b03a40a1c97baa5ed7606a1e088c889a8547d40ebda47e33d75"} Oct 30 04:59:35.439389 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:35.439326 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podStartSLOduration=1.439311149 podStartE2EDuration="1.439311149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:35.437066392 +0000 UTC m=+948.406561381" watchObservedRunningTime="2026-10-30 04:59:35.439311149 +0000 UTC m=+948.408806138" Oct 30 04:59:35.579237 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:35.579006 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:36.290247 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.290203 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:36.290881 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.290847 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:36.292254 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.292213 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" Oct 30 04:59:36.823281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.804117 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 04:59:36.823281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.810644 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.908627 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.894423 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940254 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940468 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940577 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940658 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940703 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940760 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940790 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnx5v\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940837 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:36.943119 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:36.940867 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042629 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042592 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042661 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042717 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042795 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042753 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042923 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042810 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042923 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042837 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx5v\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.042923 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042875 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043063 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.042921 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043063 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043041 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043175 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043154 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043274 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043358 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043336 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043628 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043599 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.043746 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.043723 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.045291 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.045269 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.045487 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.045464 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.062726 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.062696 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.062848 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.062749 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx5v\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.210129 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.210096 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:37.523056 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:37.523021 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f2ade0_ac3a_41bd_a4b8_867656f8e9e6.slice/crio-1ee3a48ef1b5f00191230318043c892573a2e306b380aed2e931b1c4765a6dc2 WatchSource:0}: Error finding container 1ee3a48ef1b5f00191230318043c892573a2e306b380aed2e931b1c4765a6dc2: Status 404 returned error can't find the container with id 1ee3a48ef1b5f00191230318043c892573a2e306b380aed2e931b1c4765a6dc2 Oct 30 04:59:37.524250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.524211 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 04:59:37.526059 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.526023 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:37.526148 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.526102 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:37.526201 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:37.526144 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:38.453999 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:38.453960 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" event={"ID":"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6","Type":"ContainerStarted","Data":"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc"} Oct 30 04:59:38.454210 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:38.454025 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" event={"ID":"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6","Type":"ContainerStarted","Data":"1ee3a48ef1b5f00191230318043c892573a2e306b380aed2e931b1c4765a6dc2"} Oct 30 04:59:38.558265 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:38.525290 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podStartSLOduration=2.5252671209999997 podStartE2EDuration="2.525267121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:38.509613353 +0000 UTC m=+951.479108342" watchObservedRunningTime="2026-10-30 04:59:38.525267121 +0000 UTC m=+951.494762112" Oct 30 04:59:39.210717 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:39.210682 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:39.212424 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:39.212396 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:39.212546 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:39.212446 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:40.217134 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:40.216648 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:40.217134 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:40.216716 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:41.211183 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:41.211118 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:41.211402 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:41.211192 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:42.211279 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:42.211244 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:42.211868 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:42.211842 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:43.211290 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:43.211254 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:43.211714 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:43.211323 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:44.211304 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:44.211269 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 04:59:44.211725 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:44.211324 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 04:59:44.567518 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:44.567408 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:44.580434 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:44.580403 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:45.504469 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:45.504432 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:45.504994 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:45.504883 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:45.506646 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:45.506615 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 04:59:45.528762 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:45.528724 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 04:59:48.384308 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.384260 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 04:59:48.389992 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.389967 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.408851 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.408822 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 04:59:48.417699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.417665 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"distinct-2-w1-cff4ac-kserve-self-signed-certs\"" Oct 30 04:59:48.522960 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.522907 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.522960 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.522963 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.523250 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.523101 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.523314 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.523272 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.523373 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.523338 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.523479 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.523461 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.629260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.629199 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.629530 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.629507 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.629672 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.629658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.629887 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.629874 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.630040 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.630027 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.630240 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.630208 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.630793 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.630772 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.637190 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.634527 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.638622 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.638597 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.642551 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.642525 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.650354 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.644864 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.677253 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.674950 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\") pod \"distinct-2-w1-cff4ac-kserve-6d959867b-w77t5\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:48.713597 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:48.713559 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:49.018830 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:49.018800 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f1263b_5f14_478a_9174_d7798ce3a59d.slice/crio-5e5a7706e0aff87f2be7f42eb63a99c75994247f21a4466b4c4d1f3ba6c5a314 WatchSource:0}: Error finding container 5e5a7706e0aff87f2be7f42eb63a99c75994247f21a4466b4c4d1f3ba6c5a314: Status 404 returned error can't find the container with id 5e5a7706e0aff87f2be7f42eb63a99c75994247f21a4466b4c4d1f3ba6c5a314 Oct 30 04:59:49.019493 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:49.019471 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 04:59:49.195512 ip-10-0-139-228 kubenswrapper[5058]: E1030 04:59:49.195467 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 04:59:49.649286 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:49.649255 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" event={"ID":"35f1263b-5f14-478a-9174-d7798ce3a59d","Type":"ContainerStarted","Data":"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb"} Oct 30 04:59:49.649699 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:49.649294 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" event={"ID":"35f1263b-5f14-478a-9174-d7798ce3a59d","Type":"ContainerStarted","Data":"5e5a7706e0aff87f2be7f42eb63a99c75994247f21a4466b4c4d1f3ba6c5a314"} Oct 30 04:59:49.690190 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:49.690136 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podStartSLOduration=1.6901223170000002 podStartE2EDuration="1.690122317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:49.689518747 +0000 UTC m=+962.659013737" watchObservedRunningTime="2026-10-30 04:59:49.690122317 +0000 UTC m=+962.659617381" Oct 30 04:59:49.714628 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:49.714591 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:51.987716 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:51.987676 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 04:59:51.992295 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:51.992265 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.013932 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.013901 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 04:59:52.085553 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085515 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085738 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085561 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085738 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085592 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085743 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbws\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085821 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085794 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085920 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085897 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.085956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.085944 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.086028 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.086013 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.086072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.086059 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187403 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187357 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbws\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187422 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187473 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187501 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187526 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187652 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187630 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187909 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187676 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187909 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187711 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.187909 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187744 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.188072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.187937 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.188072 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.188014 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.188176 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.188086 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.188253 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.188181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.188552 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.188529 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.189881 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.189861 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.190111 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.190091 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.207495 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.207461 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.207642 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.207524 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbws\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.262656 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.262572 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 04:59:52.267956 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.267931 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.289095 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.289060 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 04:59:52.308066 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.308033 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:52.399993 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.399831 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.400143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.400026 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.400143 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.400092 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppr2m\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.400347 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.400165 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.404283 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.400203 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.404412 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.404388 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.404470 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.404449 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.404519 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.404507 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.404575 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.404545 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.505857 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.505819 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506052 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.505872 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506052 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.505960 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppr2m\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506052 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506029 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506239 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506073 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506239 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506147 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506239 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506189 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506400 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506256 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506400 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506295 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506586 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506561 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506637 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506612 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506854 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506829 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506931 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506863 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.506931 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.506875 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.508565 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.508541 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.508794 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.508771 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.526498 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.526438 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.526498 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.526467 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppr2m\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.585733 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.585704 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:52.633781 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.633749 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 04:59:52.634149 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:52.634123 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d71ee9_d4d7_4ccf_b931_cc47bdf7fb48.slice/crio-d499c2257a75004d5b98dba183f865eb5ad9eefb109dd077ec9643062e38a156 WatchSource:0}: Error finding container d499c2257a75004d5b98dba183f865eb5ad9eefb109dd077ec9643062e38a156: Status 404 returned error can't find the container with id d499c2257a75004d5b98dba183f865eb5ad9eefb109dd077ec9643062e38a156 Oct 30 04:59:52.636835 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.636803 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.636929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.636867 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.636929 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.636898 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.674378 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.674334 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" event={"ID":"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48","Type":"ContainerStarted","Data":"d499c2257a75004d5b98dba183f865eb5ad9eefb109dd077ec9643062e38a156"} Oct 30 04:59:52.955572 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.955538 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 04:59:52.955960 ip-10-0-139-228 kubenswrapper[5058]: W1030 04:59:52.955940 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4be6d48_39d7_4b6c_9757_92bfa08c02e9.slice/crio-160f2e848b11b6cc28de11ecf9b172845855c52e9fa6cdc52ab8aa54dfe6757b WatchSource:0}: Error finding container 160f2e848b11b6cc28de11ecf9b172845855c52e9fa6cdc52ab8aa54dfe6757b: Status 404 returned error can't find the container with id 160f2e848b11b6cc28de11ecf9b172845855c52e9fa6cdc52ab8aa54dfe6757b Oct 30 04:59:52.958809 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.958769 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.958883 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.958863 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.958933 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:52.958916 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:53.683753 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:53.683255 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" event={"ID":"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48","Type":"ContainerStarted","Data":"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3"} Oct 30 04:59:53.689599 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:53.688717 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" event={"ID":"e4be6d48-39d7-4b6c-9757-92bfa08c02e9","Type":"ContainerStarted","Data":"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861"} Oct 30 04:59:53.689599 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:53.688797 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" event={"ID":"e4be6d48-39d7-4b6c-9757-92bfa08c02e9","Type":"ContainerStarted","Data":"160f2e848b11b6cc28de11ecf9b172845855c52e9fa6cdc52ab8aa54dfe6757b"} Oct 30 04:59:53.763238 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:53.763054 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podStartSLOduration=2.763035409 podStartE2EDuration="2.763035409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.737462972 +0000 UTC m=+966.706957962" watchObservedRunningTime="2026-10-30 04:59:53.763035409 +0000 UTC m=+966.732530399" Oct 30 04:59:53.788481 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:53.788428 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podStartSLOduration=1.788414028 podStartE2EDuration="1.788414028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.787812456 +0000 UTC m=+966.757307446" watchObservedRunningTime="2026-10-30 04:59:53.788414028 +0000 UTC m=+966.757909017" Oct 30 04:59:54.309092 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.309050 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 04:59:54.313281 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.313234 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:54.313434 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.313285 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:54.585944 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.585853 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 04:59:54.588504 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.588474 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:54.588661 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:54.588524 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:55.308859 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:55.308815 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:55.309369 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:55.308869 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:55.586603 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:55.586457 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:55.586603 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:55.586522 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:56.309524 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:56.309473 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:56.309948 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:56.309547 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:56.586801 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:56.586721 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:56.586801 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:56.586775 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:57.313242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:57.313150 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:57.313753 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:57.313241 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:57.602302 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:57.586413 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:57.602302 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:57.586485 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:58.322242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:58.311672 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:58.322242 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:58.311738 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:58.587303 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:58.586672 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:58.587303 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:58.586750 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:58.754245 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:58.728516 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:59.118241 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.115108 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 04:59:59.491260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.483736 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 04:59:59.491260 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.483809 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 04:59:59.598424 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.598383 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 04:59:59.604609 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.604568 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 04:59:59.608262 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.606923 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.750474 ip-10-0-139-228 kubenswrapper[5058]: I1030 04:59:59.750370 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:00:00.309144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:00.309106 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:00:00.309372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:00.309166 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:00:00.746253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:00.741140 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 05:00:00.747385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:00.747333 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 05:00:00.750888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:00.750864 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 05:00:02.292527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:02.292494 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 05:00:02.293419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:02.293346 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 05:00:02.294742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:02.294714 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 05:00:06.807745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.807701 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:00:06.812284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.812261 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.835505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.835474 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:00:06.900789 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.900755 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9c7\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.900969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.900897 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.900969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.900920 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.900989 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.901105 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.901136 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.901197 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.901312 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:06.901328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:06.901336 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.002765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002669 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.002765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002711 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.002765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002731 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002784 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002820 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.002982 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003106 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003134 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003484 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003717 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9c7\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003899 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.003998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.003911 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.004164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.004135 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.004326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.004287 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.006291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.006236 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.006291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.006282 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.027693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.027662 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9c7\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.027844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.027671 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.128696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.128592 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:07.469447 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:00:07.469414 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5e80c4_0c2b_4a1a_9ee0_5159a68368dd.slice/crio-51ebf407e46c0ef1bf3f41d086f94a748c200aaabe316ab13f27019b5f8942c2 WatchSource:0}: Error finding container 51ebf407e46c0ef1bf3f41d086f94a748c200aaabe316ab13f27019b5f8942c2: Status 404 returned error can't find the container with id 51ebf407e46c0ef1bf3f41d086f94a748c200aaabe316ab13f27019b5f8942c2 Oct 30 05:00:07.469949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.469929 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:00:07.472525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.472462 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:07.473117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.473037 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:07.473409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.473364 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:07.898747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.898711 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" event={"ID":"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd","Type":"ContainerStarted","Data":"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23"} Oct 30 05:00:07.899137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.898798 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" event={"ID":"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd","Type":"ContainerStarted","Data":"51ebf407e46c0ef1bf3f41d086f94a748c200aaabe316ab13f27019b5f8942c2"} Oct 30 05:00:07.970001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:07.969938 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podStartSLOduration=1.969923686 podStartE2EDuration="1.969923686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:07.953939363 +0000 UTC m=+980.923434352" watchObservedRunningTime="2026-10-30 05:00:07.969923686 +0000 UTC m=+980.939418675" Oct 30 05:00:08.147259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:08.145638 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:08.149446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:08.149356 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:08.149590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:08.149457 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:09.141243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:09.132324 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:09.141243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:09.132390 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:10.146244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:10.132668 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:10.146244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:10.132729 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:11.135727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:11.135686 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:11.136019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:11.135988 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:12.147242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:12.146665 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:12.147242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:12.146733 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:13.142243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:13.131662 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:13.142243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:13.131729 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:14.138244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:14.137246 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:00:14.138244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:14.137314 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 05:00:15.134612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:15.134579 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:15.135015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:15.134993 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:15.135843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:15.135820 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:00:19.191091 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:00:19.191053 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:00:21.721976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.721936 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:00:21.727719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.727690 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.745156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.745123 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:00:21.795265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795202 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795277 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795357 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795399 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkrq\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795486 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795543 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795620 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795679 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.795726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.795705 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897412 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897508 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897552 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897607 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897637 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897821 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.897966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897939 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.897958 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898175 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898285 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkrq\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898319 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898175 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898086 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.898524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.898456 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.899941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.899921 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.900338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.900320 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.918767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.918738 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:21.918945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:21.918923 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkrq\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:22.046318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:22.046210 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:22.359411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:22.359382 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:00:22.360789 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:00:22.360756 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b6dc71_f1f7_46d3_bc0f_d43fa658c968.slice/crio-e08f52817a7b82fc6a5ebe9aa3c0abd9c76c4b2b4f3c84ff4ceecbc5a50d8de6 WatchSource:0}: Error finding container e08f52817a7b82fc6a5ebe9aa3c0abd9c76c4b2b4f3c84ff4ceecbc5a50d8de6: Status 404 returned error can't find the container with id e08f52817a7b82fc6a5ebe9aa3c0abd9c76c4b2b4f3c84ff4ceecbc5a50d8de6 Oct 30 05:00:22.363680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:22.363644 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:22.363796 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:22.363724 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:22.363796 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:22.363775 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:23.072241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:23.069284 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" event={"ID":"66b6dc71-f1f7-46d3-bc0f-d43fa658c968","Type":"ContainerStarted","Data":"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001"} Oct 30 05:00:23.072241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:23.069351 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" event={"ID":"66b6dc71-f1f7-46d3-bc0f-d43fa658c968","Type":"ContainerStarted","Data":"e08f52817a7b82fc6a5ebe9aa3c0abd9c76c4b2b4f3c84ff4ceecbc5a50d8de6"} Oct 30 05:00:23.151236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:23.151139 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podStartSLOduration=2.151117782 podStartE2EDuration="2.151117782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:23.129611079 +0000 UTC m=+996.099106072" watchObservedRunningTime="2026-10-30 05:00:23.151117782 +0000 UTC m=+996.120612772" Oct 30 05:00:24.049987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:24.049757 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:24.054519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:24.054486 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:24.055331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:24.055301 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:25.046808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:25.046771 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:25.047457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:25.047407 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:26.070244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.069383 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:26.117244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.069467 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:26.322163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.317370 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.543449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.543383 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.725737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.725662 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.728387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:26.728341 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:27.059959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.059854 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:27.060245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.060185 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:27.738243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.737315 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:27.738243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.737805 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:00:27.776243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.739745 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:27.776243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:27.740094 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:00:28.009600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.009514 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:00:28.045929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.043477 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:00:28.045929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.043535 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:00:28.045929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.043622 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.127259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.054671 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:00:28.127259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.062428 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:28.127259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.111548 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:28.127259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.062775 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"distinct-2-w3-889ca9-kserve-self-signed-certs\"" Oct 30 05:00:28.262312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.246289 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.270374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.270332 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.330874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.315529 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.330874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.315659 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.330874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.315776 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.330874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.315816 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9jh\" (UniqueName: \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.418646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.418589 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.418852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.418673 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9jh\" (UniqueName: \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.418852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.418816 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.418934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.418867 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.418934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.418928 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.419016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.419004 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.419554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.419497 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.419767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.419749 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.420320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.420299 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.423882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.423849 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.424024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.423970 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.455254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.452970 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9jh\" (UniqueName: \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\") pod \"distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.477242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.477169 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:28.811813 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:00:28.811782 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a39c02e_7ca4_489a_b0cf_459359240c90.slice/crio-2351597dfe9ace4ddb0ccfa2459d0e0cef8086f32d5b5bce25b9818a22446b07 WatchSource:0}: Error finding container 2351597dfe9ace4ddb0ccfa2459d0e0cef8086f32d5b5bce25b9818a22446b07: Status 404 returned error can't find the container with id 2351597dfe9ace4ddb0ccfa2459d0e0cef8086f32d5b5bce25b9818a22446b07 Oct 30 05:00:28.812344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:28.812320 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:00:29.053061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.052630 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:29.053061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.052694 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:29.405635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.405593 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" event={"ID":"5a39c02e-7ca4-489a-b0cf-459359240c90","Type":"ContainerStarted","Data":"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae"} Oct 30 05:00:29.405811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.405649 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" event={"ID":"5a39c02e-7ca4-489a-b0cf-459359240c90","Type":"ContainerStarted","Data":"2351597dfe9ace4ddb0ccfa2459d0e0cef8086f32d5b5bce25b9818a22446b07"} Oct 30 05:00:29.455019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.454951 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podStartSLOduration=2.454925337 podStartE2EDuration="2.454925337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:29.449151405 +0000 UTC m=+1002.418646395" watchObservedRunningTime="2026-10-30 05:00:29.454925337 +0000 UTC m=+1002.424420327" Oct 30 05:00:29.477538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:29.477500 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:30.047240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:30.046604 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:00:30.047240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:30.046667 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:00:31.328668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:31.328282 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:31.329000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:31.328849 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:31.330009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:31.329988 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:00:38.477826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:38.477792 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:38.491149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:38.491112 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:38.613467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:38.613438 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:00:41.737346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:41.737299 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.029568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:42.028310 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.029830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:42.029801 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.290313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:42.283239 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:00:42.522365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:42.521115 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.610591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:42.609498 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:00:43.032769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.032567 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.060275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.033961 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:00:43.098341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.098304 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.098894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.098866 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:00:43.379973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.379567 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:00:43.750066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.750028 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:00:43.788868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.786550 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:00:43.788868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.786913 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:00:43.799975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:43.799550 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:00:49.191212 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:00:49.191177 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:00:52.162799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.162765 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:00:52.168634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.168614 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.187568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.187537 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:00:52.292587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.292545 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.292762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.292616 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qk5x\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.292762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.292744 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.292876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.292810 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.293094 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.293069 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.293177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.293113 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.293272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.293204 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.293338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.293277 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.293398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.293359 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.296373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.296345 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:00:52.301079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.301060 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.329832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.329798 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:00:52.395089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395052 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qk5x\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395112 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395133 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395149 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395184 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395230 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395301 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395388 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395446 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395540 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395563 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84r6\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395604 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395618 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395611 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.395821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395722 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395748 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395766 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395799 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395839 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.395998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.395899 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.396107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.396063 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.396324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.396301 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.396657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.396635 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.398073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.398052 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.398403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.398381 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.415055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.414999 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qk5x\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.415158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.415136 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.483109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.483074 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:52.496908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.496878 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.496929 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.496956 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84r6\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.496999 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497245 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497277 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497321 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497336 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497393 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497438 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497597 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497741 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.497820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.497785 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.498061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.498031 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.499654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.499630 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.499757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.499686 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.517360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.517334 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84r6\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.518572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.518544 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.622747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.622710 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:52.796116 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:00:52.796088 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a37256_910f_486f_855b_fbb2e850315d.slice/crio-2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb WatchSource:0}: Error finding container 2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb: Status 404 returned error can't find the container with id 2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb Oct 30 05:00:52.797669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.797640 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:00:52.798866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.798835 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.798954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.798903 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.798954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.798943 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.931440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.930447 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:00:52.933600 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:00:52.933571 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d00444_0a9b_464c_a0d3_d2f0b300c7b3.slice/crio-b4bfa5fdba4650631f7b988cfede0ab8bce94bedcf480e2111a91a34290ddb9f WatchSource:0}: Error finding container b4bfa5fdba4650631f7b988cfede0ab8bce94bedcf480e2111a91a34290ddb9f: Status 404 returned error can't find the container with id b4bfa5fdba4650631f7b988cfede0ab8bce94bedcf480e2111a91a34290ddb9f Oct 30 05:00:52.936756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.936714 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.936856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.936782 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.936856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:52.936814 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:53.013143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.013111 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" event={"ID":"17d00444-0a9b-464c-a0d3-d2f0b300c7b3","Type":"ContainerStarted","Data":"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d"} Oct 30 05:00:53.013342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.013176 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" event={"ID":"17d00444-0a9b-464c-a0d3-d2f0b300c7b3","Type":"ContainerStarted","Data":"b4bfa5fdba4650631f7b988cfede0ab8bce94bedcf480e2111a91a34290ddb9f"} Oct 30 05:00:53.015326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.015293 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" event={"ID":"a2a37256-910f-486f-855b-fbb2e850315d","Type":"ContainerStarted","Data":"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45"} Oct 30 05:00:53.015465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.015352 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" event={"ID":"a2a37256-910f-486f-855b-fbb2e850315d","Type":"ContainerStarted","Data":"2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb"} Oct 30 05:00:53.060877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.060820 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podStartSLOduration=1.060806886 podStartE2EDuration="1.060806886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:53.057718135 +0000 UTC m=+1026.027213117" watchObservedRunningTime="2026-10-30 05:00:53.060806886 +0000 UTC m=+1026.030301872" Oct 30 05:00:53.493244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.487664 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:00:53.493244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.491334 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:53.493244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:53.491391 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:54.084566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.084507 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podStartSLOduration=2.084491942 podStartE2EDuration="2.084491942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:54.081533043 +0000 UTC m=+1027.051028041" watchObservedRunningTime="2026-10-30 05:00:54.084491942 +0000 UTC m=+1027.053986930" Oct 30 05:00:54.484204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.484166 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.484401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.484256 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:54.622869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.622836 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:00:54.635434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.631356 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.635434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:54.631431 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:00:55.484756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:55.484714 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.485029 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:55.484791 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:55.630241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:55.623854 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.630241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:55.623910 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:00:56.483969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:56.483926 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.484178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:56.483992 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:56.623111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:56.623069 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.623319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:56.623125 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:00:57.485685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:57.485645 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.486318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:57.486286 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:57.623088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:57.623059 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.623355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:57.623325 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:00:58.483579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:58.483540 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.483796 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:58.483591 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:58.623491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:58.623457 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.623983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:58.623701 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:00:59.484213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:59.484176 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.484408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:59.484253 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:00:59.623437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:59.623400 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.623615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:00:59.623467 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:01:00.629123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.629086 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:01:00.629577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.629556 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:01:00.630154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.630134 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:01:00.642036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.642008 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:01:00.642347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.642322 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:01:00.644158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:00.644135 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:01:04.789268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:04.789198 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.061430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.061322 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.520093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.520046 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.593243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.591319 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.615360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.615306 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:01:05.869666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.866313 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.869666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.866977 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:05.894117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.893321 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.916038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:05.894583 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:01:06.508417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:06.508386 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:06.535990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:06.535934 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:06.537176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:06.537151 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:06.617263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:06.601863 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:07.044247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.040762 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:01:07.080066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.069959 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.119909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.115793 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.139824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.131357 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.140157 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.140126 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.140491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.140459 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.140695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.140678 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.140814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.140799 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.140935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.140896 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzcp\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.141058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.141033 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.141196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.141184 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.146780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.146754 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:01:07.152860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.152685 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:01:07.160011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.159906 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.170361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.170342 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:01:07.243008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.242978 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243020 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243136 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243177 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243374 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243419 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243443 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243499 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243641 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243672 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243694 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.243732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243727 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243889 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmqt\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.243986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.243953 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244066 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.244267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244102 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244128 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzcp\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244201 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.244502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244304 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244304 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244417 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244502 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.244680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.244660 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.246086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.246059 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.246398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.246377 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.265197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.265163 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.265324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.265275 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzcp\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.345750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.345658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.345750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.345717 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.345981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.345840 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.345981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.345887 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.345981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.345939 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmqt\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346033 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346072 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346128 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346157 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346238 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346280 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346461 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.346749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.346722 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.347119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.347098 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.348310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.348283 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.348552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.348532 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.368704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.368675 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.368862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.368715 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmqt\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.462255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.462103 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:07.477766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.477558 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:07.551179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.551144 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:01:07.575531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.575498 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.579415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.579387 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:01:07.597042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.592293 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:01:07.597042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.592412 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.597460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.597434 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:01:07.653477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.653150 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.653477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.653286 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.653477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.653314 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.653477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.653362 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.654476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.654439 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.654612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.654499 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.654687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.654605 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8sr\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.654751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.654703 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.654957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.654929 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765149 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765251 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrd8\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765391 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765454 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765510 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765612 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765694 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765728 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765754 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765784 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765832 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765908 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765933 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.765979 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.766017 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.766054 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.766990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.766105 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8sr\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.767990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.767973 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.768254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.768195 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.768702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.768675 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.768826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.768704 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.768885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.768870 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.769286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.769268 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.770784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.770765 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.792145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.792116 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.792527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.792509 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8sr\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.810342 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:01:07.810313 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278896e4_e88d_419e_97b2_31bcef5426ee.slice/crio-b86fe3f7aa5241e2b5f99a8a84195d82ca40fabbd01a3776912625251cc1e539 WatchSource:0}: Error finding container b86fe3f7aa5241e2b5f99a8a84195d82ca40fabbd01a3776912625251cc1e539: Status 404 returned error can't find the container with id b86fe3f7aa5241e2b5f99a8a84195d82ca40fabbd01a3776912625251cc1e539 Oct 30 05:01:07.811128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.811101 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:01:07.812933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.812903 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.813010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.812964 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.813010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.812999 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.835459 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:01:07.835429 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd5a154_07d7_46f3_a114_6c8904afbf51.slice/crio-871fa8008afdf724e493cea04350c79560e98c2c58f7ad1d19dcfbcdba81ce1f WatchSource:0}: Error finding container 871fa8008afdf724e493cea04350c79560e98c2c58f7ad1d19dcfbcdba81ce1f: Status 404 returned error can't find the container with id 871fa8008afdf724e493cea04350c79560e98c2c58f7ad1d19dcfbcdba81ce1f Oct 30 05:01:07.837132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.837106 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:01:07.838677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.838639 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.838782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.838727 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.838782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.838771 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:07.867954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.867924 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.867967 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.867987 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868023 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868177 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrd8\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868232 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868272 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868323 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868376 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.868873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.868787 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.869880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.869017 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.869880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.869034 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.869880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.869660 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.870152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.869903 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.872021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.872000 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.872145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.872128 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.894067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.894042 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrd8\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.898656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.898628 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:07.913499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.913460 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:07.919336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:07.918917 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:08.345078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.344467 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:01:08.345972 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:01:08.345948 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be6a7f3_6203_4b37_ac7c_4d330075ebbd.slice/crio-1ffd1702da735860c682f0d73afc77101fb8867a2beda8a4e828504f592c2b4b WatchSource:0}: Error finding container 1ffd1702da735860c682f0d73afc77101fb8867a2beda8a4e828504f592c2b4b: Status 404 returned error can't find the container with id 1ffd1702da735860c682f0d73afc77101fb8867a2beda8a4e828504f592c2b4b Oct 30 05:01:08.349303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.348574 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.349303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.348663 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.349303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.348710 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.365849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.365700 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:01:08.365941 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:01:08.365842 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123f9d88_19c1_48e0_b756_5d369cf0ea44.slice/crio-ed95c35ec69e491bd2782eb1c4d73fe0505456e638aa25fabb7805e96482b81e WatchSource:0}: Error finding container ed95c35ec69e491bd2782eb1c4d73fe0505456e638aa25fabb7805e96482b81e: Status 404 returned error can't find the container with id ed95c35ec69e491bd2782eb1c4d73fe0505456e638aa25fabb7805e96482b81e Oct 30 05:01:08.371114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.371075 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.371250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.371171 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.371322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.371212 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.532535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.532478 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" event={"ID":"278896e4-e88d-419e-97b2-31bcef5426ee","Type":"ContainerStarted","Data":"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013"} Oct 30 05:01:08.532704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.532572 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" event={"ID":"278896e4-e88d-419e-97b2-31bcef5426ee","Type":"ContainerStarted","Data":"b86fe3f7aa5241e2b5f99a8a84195d82ca40fabbd01a3776912625251cc1e539"} Oct 30 05:01:08.600526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.600444 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podStartSLOduration=2.600425911 podStartE2EDuration="2.600425911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:08.598808044 +0000 UTC m=+1041.568303060" watchObservedRunningTime="2026-10-30 05:01:08.600425911 +0000 UTC m=+1041.569920903" Oct 30 05:01:08.601017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.600975 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" event={"ID":"3be6a7f3-6203-4b37-ac7c-4d330075ebbd","Type":"ContainerStarted","Data":"1ffd1702da735860c682f0d73afc77101fb8867a2beda8a4e828504f592c2b4b"} Oct 30 05:01:08.604132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.603766 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" event={"ID":"123f9d88-19c1-48e0-b756-5d369cf0ea44","Type":"ContainerStarted","Data":"ed95c35ec69e491bd2782eb1c4d73fe0505456e638aa25fabb7805e96482b81e"} Oct 30 05:01:08.626931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.622267 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" event={"ID":"9dd5a154-07d7-46f3-a114-6c8904afbf51","Type":"ContainerStarted","Data":"d25723c7cef81106d25494d08c2d9eb1f694a97f425923cd28eaa8fc2a7eacb7"} Oct 30 05:01:08.626931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.622333 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" event={"ID":"9dd5a154-07d7-46f3-a114-6c8904afbf51","Type":"ContainerStarted","Data":"871fa8008afdf724e493cea04350c79560e98c2c58f7ad1d19dcfbcdba81ce1f"} Oct 30 05:01:08.732239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:08.687937 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podStartSLOduration=1.687918268 podStartE2EDuration="1.687918268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:08.686689117 +0000 UTC m=+1041.656184106" watchObservedRunningTime="2026-10-30 05:01:08.687918268 +0000 UTC m=+1041.657413259" Oct 30 05:01:09.466691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.466609 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:09.474240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.468594 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:09.474240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.468657 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:09.482751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.482524 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:09.482751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.482648 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:09.482751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.482689 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:09.696122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.696085 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" event={"ID":"3be6a7f3-6203-4b37-ac7c-4d330075ebbd","Type":"ContainerStarted","Data":"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da"} Oct 30 05:01:09.700421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.700307 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" event={"ID":"123f9d88-19c1-48e0-b756-5d369cf0ea44","Type":"ContainerStarted","Data":"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67"} Oct 30 05:01:09.825293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.821980 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podStartSLOduration=2.821959519 podStartE2EDuration="2.821959519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.80054722 +0000 UTC m=+1042.770042212" watchObservedRunningTime="2026-10-30 05:01:09.821959519 +0000 UTC m=+1042.791454509" Oct 30 05:01:09.869277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.868718 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podStartSLOduration=2.868694686 podStartE2EDuration="2.868694686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.864867374 +0000 UTC m=+1042.834362365" watchObservedRunningTime="2026-10-30 05:01:09.868694686 +0000 UTC m=+1042.838189678" Oct 30 05:01:09.916253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.915311 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:09.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.931321 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:09.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.931386 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:09.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.931677 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:09.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.931845 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:09.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:09.931885 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:10.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.468833 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.468907 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:10.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.482047 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.482124 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:10.921241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.920538 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.921241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.920607 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:10.921529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.921281 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.921529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:10.921327 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:11.462745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.462710 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.463009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.462983 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:11.480686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.480578 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.480686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.480631 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:11.623605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.623558 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:11.624839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.624168 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:11.745858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.745760 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:11.807243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.806309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:11.932240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.922475 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.932240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.922545 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:11.932240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.923169 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.932240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:11.923206 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:12.481190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.481143 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.481504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.481457 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:12.481998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.481965 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.485359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.482206 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:12.821305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.810293 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:12.916145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.915656 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.916145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.915723 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:12.943393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.942833 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.943393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:12.942896 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:13.513244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.487360 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:13.513244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.487435 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:13.533100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.533061 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:13.536349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.536310 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:13.550269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.540499 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:13.693243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.671604 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:13.941245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.928354 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:13.941245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:13.928433 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:14.006243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.003524 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.006243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.003602 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:14.546381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.546329 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:14.555206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.551327 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.555206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.551390 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:14.555206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.551610 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.555206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.551647 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:14.555206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.552132 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:14.614237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.614169 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:14.677919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.677882 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:14.678556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.678533 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:14.803422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.803324 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:14.809372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.809337 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:14.987702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.987666 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.988476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:14.988442 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:15.067625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.063660 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:15.132290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.125938 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.132617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.132580 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:15.471649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.471604 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.471929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.471902 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:15.489099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.489065 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.489365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.489333 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:15.522694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.522658 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:15.612647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.612586 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:15.739864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.739776 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:15.915182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.915141 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.915497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.915464 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:01:15.919601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.919520 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.919601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:15.919559 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:01:16.462766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:16.462720 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:01:16.463319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:16.462792 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:01:16.478193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:16.478149 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:01:16.478372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:16.478230 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:01:17.309242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.308902 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:17.310183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.310160 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:17.318581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.318554 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:01:17.332206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.332180 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:17.333254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.333232 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:17.337933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.337908 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:01:17.517260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.514432 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:17.517260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.515998 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:17.522417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.522362 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:01:17.749971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.749164 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:17.755378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.755308 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:17.761127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:17.761098 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:01:19.192372 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:01:19.192333 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:01:21.700417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.700382 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:01:21.706137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.706114 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.741271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.741236 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:01:21.839805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.824743 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.857598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.840104 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.857924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.857898 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.858088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.858070 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.858370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.858349 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.858541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.858522 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp25p\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.858752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.858733 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.859268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.859240 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.893404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.893363 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:21.915420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.915371 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:01:21.978956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.978867 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:01:21.979345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:21.979328 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.292658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.292569 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.309970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.309932 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.310254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.310209 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.344402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.344359 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnnr\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.344742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.344718 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.344890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.344869 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.345063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.345045 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.370344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.370308 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.433733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.347988 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.433733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.347295 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.433733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.370964 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.450626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.450586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471341 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp25p\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471407 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471487 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471523 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471641 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471714 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471775 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471871 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.483245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.471954 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.531640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.491817 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.531640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.492449 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.593731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.573416 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.594030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.593996 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.594211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.594189 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnnr\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.594380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.594361 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.599435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.599401 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.599693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.599674 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.603239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.599904 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.603485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.603459 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.614241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.603623 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.615005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.611738 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp25p\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.615498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.615465 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.629242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.615756 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.691909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.691866 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.695333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.692089 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.702287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.696128 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.805905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.719798 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.806153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.758073 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.806320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.759643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnnr\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.835243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.821586 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.836027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.835996 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.836705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.836680 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:22.870630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.870547 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:22.931280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:22.925930 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:23.107699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:23.107608 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:23.713241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:23.712412 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:01:23.758784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:23.758741 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:23.817432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:23.815318 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:23.817432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:23.815918 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.565284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.563698 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:24.867316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.779720 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.867316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.811162 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:01:24.867316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.847311 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.899243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.873878 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:24.899243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.877559 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.899243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.878042 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:24.899243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.878366 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:01:24.899243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.878443 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:01:24.954241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:24.924160 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:01:25.190144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:25.190102 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 05:01:25.693572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:25.657414 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:25.876363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:25.795358 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:01:26.237063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.207887 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:01:26.237063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.208122 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:01:26.237063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.208145 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:26.285566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.260136 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.285566 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:01:26.281212 5058 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.061s" Oct 30 05:01:26.285566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.281309 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:26.349725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.342443 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.349725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.342553 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.349725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.342627 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.821997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.815866 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.821997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.815961 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.821997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.816008 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:26.865239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:26.861623 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:01:27.256857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.256820 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.268059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.268025 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:01:27.275263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.271519 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.275263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.271452 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.311239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.291262 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.419338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.401452 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:27.419338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.401520 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:01:27.419338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.415661 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" event={"ID":"fb51a19b-b580-42cb-a7fe-30ae85662bed","Type":"ContainerStarted","Data":"55167e753112a5cc6488b493e224d06e75322cbed2a46879255a048b6f32844a"} Oct 30 05:01:27.550820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:27.550780 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" event={"ID":"322cb168-d8c6-470e-b57c-ca74debc905b","Type":"ContainerStarted","Data":"a15e6fcea3d3d279d6cedf3b9f79f30c851b13eb851e25a3a2403b703665a96c"} Oct 30 05:01:28.206505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.181546 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:28.270284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.270245 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:28.273419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.273367 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:28.299800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.299766 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:01:28.437532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.437500 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:01:28.565028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.564930 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" event={"ID":"fb51a19b-b580-42cb-a7fe-30ae85662bed","Type":"ContainerStarted","Data":"ee763c4e586ae9769f82f14998a307fdbc80739a9fdbedea45bcd2c719290211"} Oct 30 05:01:28.568296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.568257 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" event={"ID":"322cb168-d8c6-470e-b57c-ca74debc905b","Type":"ContainerStarted","Data":"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34"} Oct 30 05:01:28.618399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.618348 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podStartSLOduration=7.618334994 podStartE2EDuration="7.618334994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:28.617267414 +0000 UTC m=+1061.586762402" watchObservedRunningTime="2026-10-30 05:01:28.618334994 +0000 UTC m=+1061.587829983" Oct 30 05:01:28.670209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.670138 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podStartSLOduration=7.670115915 podStartE2EDuration="7.670115915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:28.667468293 +0000 UTC m=+1061.636963277" watchObservedRunningTime="2026-10-30 05:01:28.670115915 +0000 UTC m=+1061.639610904" Oct 30 05:01:28.931239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.931049 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:28.933042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.933010 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:28.933454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:28.933299 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:29.111241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:29.110867 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:29.111241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:29.111066 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:29.111241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:29.111115 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:29.935252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:29.934644 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:29.935252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:29.934712 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:30.108190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:30.108153 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:30.108419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:30.108246 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:30.930590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:30.930552 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:30.930875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:30.930846 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:31.112239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:31.111712 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:31.112239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:31.111777 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:31.949275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:31.945614 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:31.949275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:31.945675 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:32.108178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:32.108146 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:32.108373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:32.108198 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:32.947686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:32.947643 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:32.948381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:32.948340 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:33.117599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:33.117563 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:33.117876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:33.117847 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:33.928609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:33.928568 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:33.928890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:33.928863 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:34.108510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:34.108477 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:34.108948 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:34.108919 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:34.928326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:34.928257 5058 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:01:34.930463 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:34.930425 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:01:35.108859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:35.108826 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:01:35.109384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:35.108898 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:01:36.056841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.056805 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:36.058543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.058519 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:36.059391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.059372 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:01:36.453885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.453845 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:36.454572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.454547 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:36.456797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:36.456773 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:01:44.230380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.230342 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:01:44.236686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.236659 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.253995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.253964 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:01:44.261194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.261171 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"distinct-2-w4-d55037-kserve-self-signed-certs\"" Oct 30 05:01:44.331679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.331636 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.331906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.331884 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.331984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.331923 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.332042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.332009 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.332092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.332060 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54vx\" (UniqueName: \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.332171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.332157 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.444292 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.444362 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.444423 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.444480 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54vx\" (UniqueName: \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.445329 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.445467 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.445622 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.452774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.452687 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.454429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.453063 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.483293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.480952 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54vx\" (UniqueName: \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.483293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.481524 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.490111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.490082 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\") pod \"distinct-2-w4-d55037-kserve-56bf755658-k6czt\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.571241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.568498 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:44.959928 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:01:44.959896 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode735c001_51e4_40a2_9c01_446f39962c6f.slice/crio-5ba12fc08722937d8b54fc6cc4496f5c87cae9efdf33e5059fba6b982828f541 WatchSource:0}: Error finding container 5ba12fc08722937d8b54fc6cc4496f5c87cae9efdf33e5059fba6b982828f541: Status 404 returned error can't find the container with id 5ba12fc08722937d8b54fc6cc4496f5c87cae9efdf33e5059fba6b982828f541 Oct 30 05:01:44.960724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:44.960699 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:01:45.764841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:45.764805 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" event={"ID":"e735c001-51e4-40a2-9c01-446f39962c6f","Type":"ContainerStarted","Data":"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523"} Oct 30 05:01:45.764841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:45.764848 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" event={"ID":"e735c001-51e4-40a2-9c01-446f39962c6f","Type":"ContainerStarted","Data":"5ba12fc08722937d8b54fc6cc4496f5c87cae9efdf33e5059fba6b982828f541"} Oct 30 05:01:45.808108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:45.808037 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podStartSLOduration=1.8080226929999998 podStartE2EDuration="1.808022693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:45.807717506 +0000 UTC m=+1078.777212496" watchObservedRunningTime="2026-10-30 05:01:45.808022693 +0000 UTC m=+1078.777517717" Oct 30 05:01:46.571560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:46.571524 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:49.191718 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:01:49.191675 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:01:51.223243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.222747 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.256248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.224325 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:01:51.270005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.264309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.292241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.291005 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.385241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.384468 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:51.736244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:51.734320 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.248279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.239183 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.248279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.246435 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.248279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.247858 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:52.248279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.247945 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:01:52.322456 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.317680 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.333057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.332464 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:01:52.391427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.391381 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:52.393442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.393411 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:52.426490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.413316 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.426490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.413590 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:01:52.464101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.463282 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:01:52.692404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.683245 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:01:52.692404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.683380 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838449 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838506 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838536 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838589 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838846 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838887 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838929 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.838968 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:52.841244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:52.839041 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbsj\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.088829 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.088902 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.088951 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.088994 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.089074 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbsj\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.089141 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.089170 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.089201 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.089281 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.090678 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.090954 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.091172 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.091438 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.104301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.091953 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.247244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.243425 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.260411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.260339 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.417632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.366409 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.452785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.438331 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:53.537245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.525837 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:53.813300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.776317 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.813300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.541030 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.813300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.679735 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbsj\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.813300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.538760 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:53.862573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:53.862364 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:54.296244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.284070 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.337248 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:01:54.299360 5058 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.022s" Oct 30 05:01:54.370255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.361060 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.482611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.454864 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:01:54.649244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.597310 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:01:54.678507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.603463 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:54.801321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:54.799310 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.093726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.083432 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.288251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.286466 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.382113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.381993 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:01:55.613244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.599407 5058 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.614479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.614433 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.663246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.643305 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.685496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.685433 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:01:55.837274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:55.837209 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:01:56.300512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.295028 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:56.433821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.433093 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:01:56.654312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.654252 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:56.665260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.661620 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:01:56.690511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.690465 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:56.727606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.710972 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:56.727606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.711077 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:56.727606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.711126 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:56.846814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:56.846720 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.305499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.302937 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.473649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.473602 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.663032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.656272 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:01:57.805180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.805123 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.814989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.811282 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:01:57.814989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.812512 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:01:57.828446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.828415 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:01:57.830966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.830869 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:01:57.831874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.831813 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" event={"ID":"6183adbe-4377-4a22-a1ee-a205d5951402","Type":"ContainerStarted","Data":"49b169139a830239112864fe037f058779f8d38aaee3d55e6274634d43b70abc"} Oct 30 05:01:57.852475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:57.850682 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:01:58.858843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:58.858797 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" event={"ID":"6183adbe-4377-4a22-a1ee-a205d5951402","Type":"ContainerStarted","Data":"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f"} Oct 30 05:01:58.879798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:58.879767 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:01:58.881184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:58.881153 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:01:58.881360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:58.881212 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:01:58.905624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:58.905571 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podStartSLOduration=6.905557394 podStartE2EDuration="6.905557394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:58.90215609 +0000 UTC m=+1091.871651090" watchObservedRunningTime="2026-10-30 05:01:58.905557394 +0000 UTC m=+1091.875052432" Oct 30 05:01:59.880417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:59.880384 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:01:59.880872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:01:59.880435 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:00.879586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:00.879556 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:02:00.879773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:00.879629 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:01.879623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:01.879591 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:02:01.880149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:01.879644 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:02.880361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:02.880329 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:02:02.880757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:02.880387 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:03.879914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:03.879874 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:02:03.880101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:03.879938 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:04.584476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:04.584444 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:04.596070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:04.596036 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:04.880569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:04.880482 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:02:04.880569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:04.880550 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 05:02:05.884725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:05.884695 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:02:05.885131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:05.885102 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:02:05.885923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:05.885908 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:02:07.294034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.293997 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:02:07.298823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.298794 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.316845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.316815 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:02:07.390613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.390571 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.390801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.390650 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.390801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.390777 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.390931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.390907 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.391122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.391102 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.391192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.391141 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.391308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.391286 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.391370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.391336 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8hh\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.391409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.391392 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.492767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492729 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.492767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492772 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492825 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492861 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8hh\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492887 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.492969 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493203 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493293 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493370 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493383 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493436 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493633 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.493987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.493968 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.496009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.495985 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.496133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.496116 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.513963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.513926 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.514167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.514145 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8hh\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.615644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.615552 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:07.938043 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:02:07.938014 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49afaaf_b98e_49c2_af0e_d5316057cb6b.slice/crio-a1fb1e37d84a4c4cd4de7f3881e2c829cdbb6ae7d19b2f0f60ab25623cd165c2 WatchSource:0}: Error finding container a1fb1e37d84a4c4cd4de7f3881e2c829cdbb6ae7d19b2f0f60ab25623cd165c2: Status 404 returned error can't find the container with id a1fb1e37d84a4c4cd4de7f3881e2c829cdbb6ae7d19b2f0f60ab25623cd165c2 Oct 30 05:02:07.939990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.939967 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:02:07.940965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.940930 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:02:07.941042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.940997 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:02:07.941042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.941026 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:02:07.967762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:07.967716 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" event={"ID":"b49afaaf-b98e-49c2-af0e-d5316057cb6b","Type":"ContainerStarted","Data":"a1fb1e37d84a4c4cd4de7f3881e2c829cdbb6ae7d19b2f0f60ab25623cd165c2"} Oct 30 05:02:08.977991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:08.977948 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" event={"ID":"b49afaaf-b98e-49c2-af0e-d5316057cb6b","Type":"ContainerStarted","Data":"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8"} Oct 30 05:02:09.616682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:09.616647 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:09.618556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:09.618529 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:09.618688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:09.618587 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:10.616410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:10.616375 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:10.616816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:10.616436 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:11.616250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:11.616176 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:11.616432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:11.616260 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:12.616440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:12.616407 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:12.616935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:12.616469 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:13.616314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:13.616277 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:13.616489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:13.616346 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:14.616812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:14.616777 5058 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:02:14.619421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:14.616851 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:02:15.621372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:15.621342 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:15.621872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:15.621744 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:15.622422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:15.622403 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:02:15.694375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:15.694313 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podStartSLOduration=8.694296217 podStartE2EDuration="8.694296217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:09.051624461 +0000 UTC m=+1102.021119453" watchObservedRunningTime="2026-10-30 05:02:15.694296217 +0000 UTC m=+1108.663791206" Oct 30 05:02:19.191362 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:02:19.191323 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:02:20.740265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:20.728390 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:20.887241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:20.882789 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.316339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.316295 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.330379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.330334 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.455444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.453619 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:21.623296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.622322 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.749366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.749319 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.750130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.749999 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.751106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:21.751082 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:02:22.291250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.196565 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:22.291250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.198858 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:02:22.365275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.341597 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:02:22.365275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.342565 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:02:22.660277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.646433 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:22.879382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.872485 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:22.923670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.923563 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:22.953427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.953355 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:22.962299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:22.962267 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:02:23.075783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.003426 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:23.076346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.076320 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:02:23.199439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.199328 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:23.390501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.390453 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.540286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.540171 5058 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-9pkhh container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:02:23.540562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.540520 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9pkhh" podUID="38619267-fa1a-42a4-b54c-abf89f69ba9f" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.651873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.651822 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.887305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.878387 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.967246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:23.956775 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.088449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.087418 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.427243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.427130 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.693331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.681102 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:24.712528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.712477 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.896651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.896592 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.989307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:24.963365 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:25.077142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.077083 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.094245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.090802 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.470836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.448983 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.700370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.700317 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.749596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.749453 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 05:02:25.770744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.770686 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.917244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.910580 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:02:25.941244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.938265 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:02:25.964367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:25.964308 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.109309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.108464 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:26.501303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.495990 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:02:26.706412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.706360 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:26.913244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.912347 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:02:26.973243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.972841 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.001243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:26.973835 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.220311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.216762 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:27.518300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.499643 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.708448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.708394 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.930050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.929967 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.023246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.995593 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.023246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:27.999615 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:28.265294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:28.247721 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.551962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:28.551865 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.745744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:28.744291 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.935405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:28.935358 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.998395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:28.997428 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:29.006892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.005287 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:29.045266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.044249 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:02:29.079049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.079010 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:29.287293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.259780 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:29.304127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.304085 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:02:29.304570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.304549 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:02:29.327588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.326749 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:02:29.379293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:29.375872 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:02:48.875362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:48.875315 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.332969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.332384 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.448493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.448452 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.500289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.480382 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.668450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.668398 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.736244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.718759 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.845242 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:02:49.841454 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:02:49.876598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.876540 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.886253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:49.878314 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:02:50.243243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.230911 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.243442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.243234 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:50.359245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.344295 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.359245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.344709 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:02:50.553425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.547361 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.559554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.554963 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:02:50.714246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.683351 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.817425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.817335 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:50.891058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:50.891015 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:02:51.256245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.244176 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.261313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.261269 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:51.390680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.390630 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:51.549313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.544276 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:02:51.549313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.544373 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:02:51.585244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.579002 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.701456 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.701397 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.000340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.896692 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.001804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:51.909424 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.297781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.297678 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.303243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.299620 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.399853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.398191 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.572324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.564742 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:02:52.572324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.564806 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.598243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.597170 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.723766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:52.713156 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:02:53.081243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.064642 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.309694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.305191 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.338296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.337955 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:53.434258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.403885 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.622333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.609253 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:02:53.728255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:53.728184 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.106693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.106650 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:02:54.337249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.334143 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.341333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.341288 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.419105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.419045 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.630383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.630332 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:02:54.732503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:54.732380 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.144368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.132346 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.144368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.133396 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.337332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.337283 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.346633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.346587 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.426270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.426150 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.599246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.582320 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.599246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.582594 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 30 05:02:55.645114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.645070 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.785818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:55.785724 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:02:56.140311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.134896 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.381258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.369682 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.381258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.370006 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:56.640128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.612391 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:56.780243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.741863 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.917312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:56.914895 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:57.138456 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.138400 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:57.677243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.635580 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:02:57.853276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.649434 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:02:57.893243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.889415 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:02:57.893243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.890867 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:57.923243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:57.918288 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:02:58.411242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.407754 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.844419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.844312 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.880489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.879586 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.895245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.891299 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.919342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.919299 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.941251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:58.920717 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:59.509132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.508257 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:59.835808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.800517 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 05:02:59.849270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.848666 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:59.871591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.871557 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:02:59.871930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.871913 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:02:59.873285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.872806 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:02:59.873285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.872858 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:02:59.873285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.872907 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:02:59.882282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:02:59.882206 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:03:03.262179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.262136 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:03:03.268349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.268317 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.289969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.289941 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:03:03.297103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.297078 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d67de78b-openshift-default-dockercfg-8csk5\"" Oct 30 05:03:03.352042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352000 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvm4\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352063 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352263 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352446 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352572 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352614 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352649 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352736 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.352906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.352830 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.518329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.509506 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.525663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.518621 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.525960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.525935 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.526093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.526073 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.526314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.526296 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.526552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.526534 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.526723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.526708 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvm4\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.526877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.526859 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.532357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.532325 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.546278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.534027 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.548742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.542947 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.599847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.543204 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.599847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.546877 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.599847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.547116 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.599847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.597924 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.605008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.604943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.622503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.622471 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.624099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.624071 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvm4\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\") pod \"e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:03.894135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:03.894102 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:04.206047 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:03:04.206005 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab6873f_867f_4bb4_8712_63aec00594a7.slice/crio-b697ee5762de208b3027fd5937f86d460fa010f17058df7a0b6363ef176284f0 WatchSource:0}: Error finding container b697ee5762de208b3027fd5937f86d460fa010f17058df7a0b6363ef176284f0: Status 404 returned error can't find the container with id b697ee5762de208b3027fd5937f86d460fa010f17058df7a0b6363ef176284f0 Oct 30 05:03:04.207553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:04.207526 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:03:04.208432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:04.208402 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:04.208504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:04.208466 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:04.208504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:04.208494 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:05.175838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:05.175801 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" event={"ID":"8ab6873f-867f-4bb4-8712-63aec00594a7","Type":"ContainerStarted","Data":"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462"} Oct 30 05:03:05.176294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:05.175865 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" event={"ID":"8ab6873f-867f-4bb4-8712-63aec00594a7","Type":"ContainerStarted","Data":"b697ee5762de208b3027fd5937f86d460fa010f17058df7a0b6363ef176284f0"} Oct 30 05:03:05.235662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:05.235599 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" podStartSLOduration=2.235584125 podStartE2EDuration="2.235584125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:05.231739336 +0000 UTC m=+1158.201234339" watchObservedRunningTime="2026-10-30 05:03:05.235584125 +0000 UTC m=+1158.205079114" Oct 30 05:03:05.895275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:05.895244 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:05.900428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:05.900402 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:06.182373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:06.182293 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:06.183298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:06.183281 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:03:08.768723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.768684 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:03:08.773354 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.773332 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:08.790953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.790918 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:03:08.798991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.798964 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d67de78b-dockercfg-66ltf\"" Oct 30 05:03:08.839250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.836506 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d67de78b\"" Oct 30 05:03:08.978575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.978536 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7v5\" (UniqueName: \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.000941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:08.982377 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.149024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.148977 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.149685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.149512 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7v5\" (UniqueName: \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.151112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.151089 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.208985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.208947 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7v5\" (UniqueName: \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\") pod \"payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.388727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.388691 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:09.693931 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:03:09.693899 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868e0713_c916_4d9f_96e4_497477384277.slice/crio-e7970634bd9d7d29d790da34a3fd76f4940435c76226ff7af2eb3e5f87761f55 WatchSource:0}: Error finding container e7970634bd9d7d29d790da34a3fd76f4940435c76226ff7af2eb3e5f87761f55: Status 404 returned error can't find the container with id e7970634bd9d7d29d790da34a3fd76f4940435c76226ff7af2eb3e5f87761f55 Oct 30 05:03:09.694753 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:09.694727 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:03:10.272781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.272746 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" event={"ID":"868e0713-c916-4d9f-96e4-497477384277","Type":"ContainerStarted","Data":"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e"} Oct 30 05:03:10.273189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.272799 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" event={"ID":"868e0713-c916-4d9f-96e4-497477384277","Type":"ContainerStarted","Data":"e7970634bd9d7d29d790da34a3fd76f4940435c76226ff7af2eb3e5f87761f55"} Oct 30 05:03:10.273189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.272878 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:10.317235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.317165 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" podStartSLOduration=2.317147389 podStartE2EDuration="2.317147389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:10.314040929 +0000 UTC m=+1163.283535919" watchObservedRunningTime="2026-10-30 05:03:10.317147389 +0000 UTC m=+1163.286642379" Oct 30 05:03:10.980176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.980138 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:03:10.985767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:10.985741 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.003768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.003737 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.004081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.004056 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.004210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.004112 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.004210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.004197 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.005261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.005198 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vgd\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.005389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.005289 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.005389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.005384 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.005626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.005445 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.005626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.005551 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.019529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.019484 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:03:11.020544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.020521 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0fba292-openshift-default-dockercfg-ql9zz\"" Oct 30 05:03:11.107132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107097 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107166 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107191 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107214 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107426 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107496 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vgd\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107539 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107615 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.107700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.107673 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.108097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.108071 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.108163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.108091 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.108163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.108120 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.108280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.108160 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.108280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.108195 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.109585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.109558 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.109788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.109772 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.129011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.128980 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.129139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.129104 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vgd\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-7r6dr\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.317289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.307963 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:11.821380 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:03:11.821350 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b083858_3553_4ad6_9f5e_eedec2a7bb24.slice/crio-80b4743171f2b6e08f0116f780c756c9a5e6cca585014f7b92384ddcf5b71f64 WatchSource:0}: Error finding container 80b4743171f2b6e08f0116f780c756c9a5e6cca585014f7b92384ddcf5b71f64: Status 404 returned error can't find the container with id 80b4743171f2b6e08f0116f780c756c9a5e6cca585014f7b92384ddcf5b71f64 Oct 30 05:03:11.823558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.823535 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:03:11.824347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.824309 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:11.824414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.824400 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:11.824460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:11.824442 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:12.363885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:12.363850 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerStarted","Data":"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5"} Oct 30 05:03:12.364524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:12.364502 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerStarted","Data":"80b4743171f2b6e08f0116f780c756c9a5e6cca585014f7b92384ddcf5b71f64"} Oct 30 05:03:12.421646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:12.421587 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" podStartSLOduration=2.42157176 podStartE2EDuration="2.42157176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:12.419897253 +0000 UTC m=+1165.389392242" watchObservedRunningTime="2026-10-30 05:03:12.42157176 +0000 UTC m=+1165.391066749" Oct 30 05:03:13.308687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:13.308653 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:13.317320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:13.317289 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:13.371060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:13.371030 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:13.372207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:13.372176 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:03:15.971726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:15.971691 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:03:15.976577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:15.976554 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:15.994017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:15.993993 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:03:15.997878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:15.997844 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:15.998063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:15.997901 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcjw\" (UniqueName: \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.042264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.032681 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b0fba292\"" Oct 30 05:03:16.042264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.032815 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b0fba292-dockercfg-p84m8\"" Oct 30 05:03:16.136242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.116184 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcjw\" (UniqueName: \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.136242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.116520 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.172244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.164635 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.260343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.238425 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcjw\" (UniqueName: \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\") pod \"payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.305950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.300548 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:16.810284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:16.810252 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:03:17.490890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:17.490833 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" event={"ID":"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13","Type":"ContainerStarted","Data":"c141bc834e3eddaed59e0967db558e545bcd7309b15f6ee6568213a28c0f2b02"} Oct 30 05:03:17.491322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:17.490920 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:17.491322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:17.490944 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" event={"ID":"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13","Type":"ContainerStarted","Data":"619932f463399afc027ec0f2e598cca4f27baeb0844dbcba1cb82c9a86d685df"} Oct 30 05:03:17.584257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:17.573981 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" podStartSLOduration=2.573960718 podStartE2EDuration="2.573960718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:17.57132576 +0000 UTC m=+1170.540820761" watchObservedRunningTime="2026-10-30 05:03:17.573960718 +0000 UTC m=+1170.543455708" Oct 30 05:03:19.545244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.531049 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:19.545244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.532632 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:03:19.566301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.559332 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.607244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.606925 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.623335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.623292 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:03:19.721903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.721858 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.722324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:19.722130 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.952265 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:03:19.947873 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:03:20.131422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.131376 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.544241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.532057 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:20.579709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.571157 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.581210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.580502 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:03:20.581631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.581613 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:03:20.615413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.615369 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:20.616168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.616141 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:03:20.644242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.625613 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.644242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.626325 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:03:20.732243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.723447 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.793786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:20.793756 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:03:21.582403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.582348 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.657242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.638755 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.724333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.724282 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:03:21.823245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.799743 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:03:21.930305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.894309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.930305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.895391 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:21.930305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.924169 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.930305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.925705 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.006284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.965072 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:22.006284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:21.965132 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.074276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.019934 5058 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:22.074591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.074552 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.666245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.647558 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:03:22.711395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.711362 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:03:22.729588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.729531 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.747823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.747786 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:22.748087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.748055 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.920453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.920353 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.018246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.925319 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.186308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.925369 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:23.235611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:22.945693 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:03:23.717023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:23.716989 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:03:23.796888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:23.752980 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:03:23.959411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:23.959362 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.963714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:23.963688 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:03:23.968710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:23.968642 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:03:24.164498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:24.164439 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:03:24.192258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:24.189891 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:03:24.391357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:24.391329 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:03:28.825833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:28.825795 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:03:37.914249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:37.913389 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:37.965417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:37.965357 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.750917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.748323 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.937795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.927584 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.937795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.928027 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:03:38.937795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.929928 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.970561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.968386 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:03:38.970561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:38.968904 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:03:39.030324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.029302 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:03:39.237244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.229344 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.483276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.480775 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.774643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.768372 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.774643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.769929 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:03:39.774643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.770987 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.970503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.954369 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.970503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.955900 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:39.970503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:39.956409 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:03:40.187945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.187893 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:03:40.194888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.194849 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:03:40.227611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.227577 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:03:40.255309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.229943 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:03:40.257099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.257070 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:03:40.871253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.835952 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:03:40.955397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.955322 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:40.969251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:40.969180 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.192392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.192333 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.246355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.246304 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.268519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.268475 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.942306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.926726 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.944650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.944607 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.957900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.957857 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.981770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:41.981723 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:42.325353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.239599 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.325353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.249426 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:03:42.325353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.293618 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.964687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.964628 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.966018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.965983 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:03:42.985315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:42.985261 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.249560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:43.249435 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.250457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:43.250382 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.325108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:43.325068 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:03:43.983276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:43.972545 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:43.984728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:43.984685 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.170428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.170385 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:44.259312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.251330 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.271256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.259737 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:03:44.333382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.333332 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.723861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.723498 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:44.723861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.723568 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.981302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.979310 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:44.993248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:44.986376 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:03:45.095239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.090338 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.185509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.185439 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:03:45.266861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.266731 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.276609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.273072 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.317773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.317735 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:45.319333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.319292 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.339323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.339283 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.535535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.535437 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:03:45.594243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.577779 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 05:03:45.712248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.670758 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.769746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.769716 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:03:45.807031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.806918 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:03:45.999306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:45.987456 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:46.301285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:46.301240 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:46.396411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:46.359326 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:03:46.433190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:46.431930 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:03:46.813834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:46.813791 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:03:47.010240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:47.008305 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:47.018646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:47.013527 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:03:47.320720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:47.316489 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:03:47.416244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:47.415835 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:03:47.431716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:47.430774 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:03:49.192014 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:03:49.191976 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:03:52.344562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.344525 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ns7rz-vvjmg"] Oct 30 05:03:52.356485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.356442 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.362893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.362863 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ns7rz-vvjmg"] Oct 30 05:03:52.372404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.372374 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wfdkq\"" Oct 30 05:03:52.417399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.417361 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.417710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.417682 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5h5\" (UniqueName: \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.417851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.417806 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.417925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.417872 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.519714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.519675 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5h5\" (UniqueName: \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.519892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.519845 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.519892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.519876 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.520057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.520023 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.520342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.520319 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.520731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.520712 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.521292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.521264 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.540899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.540865 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5h5\" (UniqueName: \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\") pod \"periodic-gathering-ns7rz-vvjmg\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.676745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.676713 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:03:52.982826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:52.977738 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ns7rz-vvjmg"] Oct 30 05:03:53.270009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:53.269865 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" event={"ID":"dbd5ac21-64cd-46ae-bea8-6047761a685d","Type":"ContainerStarted","Data":"24b3280b10f89359ef38ddd552a39212c1aea169f824553b2e2b96a84dbbeac3"} Oct 30 05:03:54.214373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:54.214337 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" event={"ID":"dbd5ac21-64cd-46ae-bea8-6047761a685d","Type":"ContainerStarted","Data":"93bee2a205abd3f56b03a04fe46584e2cde624c27e5846a38d86d7c95b471929"} Oct 30 05:03:54.263421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:54.263367 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" podStartSLOduration=2.263353472 podStartE2EDuration="2.263353472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:54.259362596 +0000 UTC m=+1207.228857587" watchObservedRunningTime="2026-10-30 05:03:54.263353472 +0000 UTC m=+1207.232848553" Oct 30 05:03:55.912739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:55.912705 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:03:55.918187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:55.918158 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:55.943227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:55.943181 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:03:55.950425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:55.950398 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-097d90-openshift-default-dockercfg-nvjb9\"" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027151 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027363 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pn9\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027430 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027513 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027723 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027775 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027813 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027873 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.045275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.027904 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.130874 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.130988 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131213 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131291 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131331 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131389 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131421 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131533 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.131586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pn9\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.132202 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.133079 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.141895 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.144247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.142164 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.167292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.150848 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.167292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.163544 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.180667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.176969 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.180667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.180136 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pn9\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.217242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.206155 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\") pod \"e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.282862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.276602 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:56.828007 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:03:56.827975 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9296f7_7aed_4117_9407_d5a43216af6a.slice/crio-25e5ac2e409ce7748137cb6fc7460bb20000a7c4f30943338fbedbdab8541ed6 WatchSource:0}: Error finding container 25e5ac2e409ce7748137cb6fc7460bb20000a7c4f30943338fbedbdab8541ed6: Status 404 returned error can't find the container with id 25e5ac2e409ce7748137cb6fc7460bb20000a7c4f30943338fbedbdab8541ed6 Oct 30 05:03:56.829556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.829532 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:03:56.830701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.830672 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:56.830775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.830731 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:56.830775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:56.830761 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:03:57.256328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:57.256289 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerStarted","Data":"eade0cae3c9157218e8898a5af77bb04f01f00e0937ec2ea3285db199c5aaddd"} Oct 30 05:03:57.256782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:57.256342 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerStarted","Data":"25e5ac2e409ce7748137cb6fc7460bb20000a7c4f30943338fbedbdab8541ed6"} Oct 30 05:03:57.277835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:57.277806 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:57.283159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:57.283135 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:57.327776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:57.327710 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podStartSLOduration=2.3276940870000002 podStartE2EDuration="2.327694087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:57.312831336 +0000 UTC m=+1210.282326325" watchObservedRunningTime="2026-10-30 05:03:57.327694087 +0000 UTC m=+1210.297189075" Oct 30 05:03:58.269157 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:58.269059 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:03:58.270974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:03:58.270949 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:04:01.688551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:01.688516 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:04:01.695857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:01.695827 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:01.711255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:01.711202 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:04:01.718107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:01.718079 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-097d90\"" Oct 30 05:04:01.729397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:01.729372 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-097d90-dockercfg-w7qnp\"" Oct 30 05:04:02.094558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.094469 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.095083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.095051 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lfc\" (UniqueName: \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.201245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.196647 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lfc\" (UniqueName: \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.201245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.196828 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.201245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.198060 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.207281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.207244 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:04:02.225981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.225947 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:04:02.226320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.226300 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.237718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.236980 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lfc\" (UniqueName: \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.298880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.298846 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.299069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.299044 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scn85\" (UniqueName: \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.318575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.318543 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:02.401001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.400791 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scn85\" (UniqueName: \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.401150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.401092 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.402984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.402578 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.421915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.421891 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scn85\" (UniqueName: \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\") pod \"payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.565691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.565662 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:02.626990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.626957 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:04:02.633107 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:02.633070 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6921b781_3da1_433d_a4f1_d94f74aca1a9.slice/crio-dbcb4cf226d94d690a603b1a32bd8884a27705b19587d14b2b1eba3fa2821ae3 WatchSource:0}: Error finding container dbcb4cf226d94d690a603b1a32bd8884a27705b19587d14b2b1eba3fa2821ae3: Status 404 returned error can't find the container with id dbcb4cf226d94d690a603b1a32bd8884a27705b19587d14b2b1eba3fa2821ae3 Oct 30 05:04:02.872564 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:02.872537 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bae227_08fd_42c3_abd0_4782180573fb.slice/crio-ee269785ad916395e64fd7a9b1d8ba415f3e1329119a026f4b046728387f81fd WatchSource:0}: Error finding container ee269785ad916395e64fd7a9b1d8ba415f3e1329119a026f4b046728387f81fd: Status 404 returned error can't find the container with id ee269785ad916395e64fd7a9b1d8ba415f3e1329119a026f4b046728387f81fd Oct 30 05:04:02.874271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:02.874194 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:04:03.434246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.431955 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" event={"ID":"24bae227-08fd-42c3-abd0-4782180573fb","Type":"ContainerStarted","Data":"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712"} Oct 30 05:04:03.434246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.432023 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" event={"ID":"24bae227-08fd-42c3-abd0-4782180573fb","Type":"ContainerStarted","Data":"ee269785ad916395e64fd7a9b1d8ba415f3e1329119a026f4b046728387f81fd"} Oct 30 05:04:03.434246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.433251 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:03.459484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.459432 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" event={"ID":"6921b781-3da1-433d-a4f1-d94f74aca1a9","Type":"ContainerStarted","Data":"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e"} Oct 30 05:04:03.461375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.461354 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:03.461670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.461627 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" event={"ID":"6921b781-3da1-433d-a4f1-d94f74aca1a9","Type":"ContainerStarted","Data":"dbcb4cf226d94d690a603b1a32bd8884a27705b19587d14b2b1eba3fa2821ae3"} Oct 30 05:04:03.520242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.514139 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" podStartSLOduration=2.51411287 podStartE2EDuration="2.51411287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:03.473631794 +0000 UTC m=+1216.443126796" watchObservedRunningTime="2026-10-30 05:04:03.51411287 +0000 UTC m=+1216.483607860" Oct 30 05:04:03.556264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:03.549995 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" podStartSLOduration=2.5499720139999997 podStartE2EDuration="2.549972014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:03.519824628 +0000 UTC m=+1216.489319621" watchObservedRunningTime="2026-10-30 05:04:03.549972014 +0000 UTC m=+1216.519467008" Oct 30 05:04:07.261362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.261309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:07.315505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.315465 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:04:07.349287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.325989 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:07.998382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.998330 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:08.000654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.999163 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:04:08.001380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:07.999194 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:08.263298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.262427 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:04:08.263298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.263272 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:04:08.348902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.348850 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:08.349609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.349584 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:04:08.359100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.359059 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:08.359827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:08.359805 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:04:09.003392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.003344 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:09.057244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.013743 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:04:09.058516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.016309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:04:09.070521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.016544 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:04:09.081884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.081851 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:04:09.082135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.082119 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:04:09.277939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.277835 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:09.371331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.359284 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:09.432242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.380791 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:04:09.716244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.662794 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 05:04:09.983304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:09.863525 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.062258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.050203 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:10.169407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.065671 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.799243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.066728 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.800246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.290592 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.800966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.379030 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.899547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.403370 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.900134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:10.660372 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:04:11.165054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.165001 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:11.761377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.761329 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:04:11.788374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.788319 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.800401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.800356 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.802027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.801997 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.885385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.885336 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:04:11.900068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:11.900026 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:04:12.176365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.176310 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.764363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.764316 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.814408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.813327 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.814408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.813797 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:04:12.959245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.899331 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:04:12.959245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:12.905313 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:04:13.180370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.180317 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:13.950241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.934093 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:04:13.996038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.948319 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:13.996556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.951346 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:04:13.998066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.951380 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:04:13.998921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:13.951416 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:14.196359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:14.196309 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:15.046392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.046326 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:15.059249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.047881 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:04:15.079702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.062372 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:04:15.084039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.064626 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:04:15.084667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.064760 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:04:15.085172 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.083532 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 05:04:15.226245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.201576 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:15.516956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.516916 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:04:15.523040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.523011 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:04:15.600735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.600695 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:15.641663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.641621 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 05:04:15.695464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.695415 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:04:15.698797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.697087 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:04:15.823904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.808110 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:04:15.823904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.808922 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:04:15.823904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.812324 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:04:15.823904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:15.817121 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:04:16.323659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.323617 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" podUID="868e0713-c916-4d9f-96e4-497477384277" containerName="payload-pre-processing" containerID="cri-o://4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e" gracePeriod=25 Oct 30 05:04:16.619751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.619707 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:04:16.890670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.890639 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:04:16.901129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.901096 5058 generic.go:356] "Generic (PLEG): container finished" podID="868e0713-c916-4d9f-96e4-497477384277" containerID="4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e" exitCode=0 Oct 30 05:04:16.901599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.901580 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" Oct 30 05:04:16.901729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.901612 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" event={"ID":"868e0713-c916-4d9f-96e4-497477384277","Type":"ContainerDied","Data":"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e"} Oct 30 05:04:16.901729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.901675 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6" event={"ID":"868e0713-c916-4d9f-96e4-497477384277","Type":"ContainerDied","Data":"e7970634bd9d7d29d790da34a3fd76f4940435c76226ff7af2eb3e5f87761f55"} Oct 30 05:04:16.901729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.901698 5058 scope.go:122] "RemoveContainer" containerID="4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e" Oct 30 05:04:16.915864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.915842 5058 scope.go:122] "RemoveContainer" containerID="4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e" Oct 30 05:04:16.916109 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:16.916090 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e\": container with ID starting with 4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e not found: ID does not exist" containerID="4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e" Oct 30 05:04:16.916207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.916116 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e"} err="failed to get container status \"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e\": rpc error: code = NotFound desc = could not find container \"4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e\": container with ID starting with 4ea2f14aace3b9096d62fb8d215d51639ee91f33d6f5b8a2011f357deaf5fd8e not found: ID does not exist" Oct 30 05:04:16.988124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.982111 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\" (UniqueName: \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\") pod \"868e0713-c916-4d9f-96e4-497477384277\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " Oct 30 05:04:16.988124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.982176 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\") pod \"868e0713-c916-4d9f-96e4-497477384277\" (UID: \"868e0713-c916-4d9f-96e4-497477384277\") " Oct 30 05:04:16.988124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:16.984338 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume" pod "868e0713-c916-4d9f-96e4-497477384277" (UID: "868e0713-c916-4d9f-96e4-497477384277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:17.097242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:17.096698 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/868e0713-c916-4d9f-96e4-497477384277-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:17.123503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:17.123433 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5" pod "868e0713-c916-4d9f-96e4-497477384277" (UID: "868e0713-c916-4d9f-96e4-497477384277"). InnerVolumeSpecName "kube-api-access-kt7v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:17.380295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:17.378720 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt7v5\" (UniqueName: \"kubernetes.io/projected/868e0713-c916-4d9f-96e4-497477384277-kube-api-access-kt7v5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:17.649624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:17.649587 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:04:17.720614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:17.720573 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerName="istio-proxy" containerID="cri-o://4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462" gracePeriod=30 Oct 30 05:04:18.269700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:18.269662 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:04:18.333243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:18.316494 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6"] Oct 30 05:04:18.354607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:18.334293 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:04:18.407244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:18.401333 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:18.464975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:18.464877 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:19.166615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:19.166568 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868e0713-c916-4d9f-96e4-497477384277" path="/var/lib/kubelet/pods/868e0713-c916-4d9f-96e4-497477384277/volumes" Oct 30 05:04:19.273287 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:19.271610 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:04:21.620735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:21.620667 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" podUID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" containerName="payload-pre-processing" containerID="cri-o://c141bc834e3eddaed59e0967db558e545bcd7309b15f6ee6568213a28c0f2b02" gracePeriod=25 Oct 30 05:04:22.068618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.068433 5058 generic.go:356] "Generic (PLEG): container finished" podID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" containerID="c141bc834e3eddaed59e0967db558e545bcd7309b15f6ee6568213a28c0f2b02" exitCode=0 Oct 30 05:04:22.068618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.068522 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" event={"ID":"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13","Type":"ContainerDied","Data":"c141bc834e3eddaed59e0967db558e545bcd7309b15f6ee6568213a28c0f2b02"} Oct 30 05:04:22.112431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.112017 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:04:22.221983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.221940 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\" (UniqueName: \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\") pod \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " Oct 30 05:04:22.222173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.222020 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\") pod \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\" (UID: \"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13\") " Oct 30 05:04:22.223196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.223167 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume" pod "778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" (UID: "778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:22.226866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.226823 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw" pod "778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" (UID: "778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13"). InnerVolumeSpecName "kube-api-access-xpcjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:22.227724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.227363 5058 patch_prober.go:29] interesting pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:04:22.227724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.227430 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:22.344314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.344135 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcjw\" (UniqueName: \"kubernetes.io/projected/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-kube-api-access-xpcjw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:22.344314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:22.344171 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:23.007117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.007059 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:04:23.008815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.008785 5058 cpu_manager.go:414] "containerMap: removing container" podUID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" containerName="payload-pre-processing" Oct 30 05:04:23.008962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.008947 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" containerName="payload-pre-processing" Oct 30 05:04:23.009078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.009066 5058 cpu_manager.go:414] "containerMap: removing container" podUID="868e0713-c916-4d9f-96e4-497477384277" containerName="payload-pre-processing" Oct 30 05:04:23.009156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.009145 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868e0713-c916-4d9f-96e4-497477384277" containerName="payload-pre-processing" Oct 30 05:04:23.009476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.009447 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" containerName="payload-pre-processing" Oct 30 05:04:23.009606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.009595 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="868e0713-c916-4d9f-96e4-497477384277" containerName="payload-pre-processing" Oct 30 05:04:23.014212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.014191 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.035522 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.035477 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5a926a9d\"" Oct 30 05:04:23.040984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.039591 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:04:23.092519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.092480 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.093184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.093149 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdm4\" (UniqueName: \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.102659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.102602 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" event={"ID":"778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13","Type":"ContainerDied","Data":"619932f463399afc027ec0f2e598cca4f27baeb0844dbcba1cb82c9a86d685df"} Oct 30 05:04:23.102824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.102725 5058 scope.go:122] "RemoveContainer" containerID="c141bc834e3eddaed59e0967db558e545bcd7309b15f6ee6568213a28c0f2b02" Oct 30 05:04:23.102919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.102899 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp" Oct 30 05:04:23.224825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.206635 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.322299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.266170 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.322299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.302346 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdm4\" (UniqueName: \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.528392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.492207 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdm4\" (UniqueName: \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\") pod \"payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.565238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.563806 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:04:23.565238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.563863 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp"] Oct 30 05:04:23.655340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:23.651712 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:23.996886 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:23.996850 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976c67e7_b1c9_423c_9699_2ad23ab5667e.slice/crio-070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8 WatchSource:0}: Error finding container 070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8: Status 404 returned error can't find the container with id 070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8 Oct 30 05:04:24.001259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.000759 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:04:24.134210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.134141 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" event={"ID":"976c67e7-b1c9-423c-9699-2ad23ab5667e","Type":"ContainerStarted","Data":"070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8"} Oct 30 05:04:24.590130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.581980 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:04:24.607766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.606588 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:04:24.607766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.606707 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.634502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.624293 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-097d90-serving-cert\"" Oct 30 05:04:24.670041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.659599 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.670041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.659790 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.670041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.659838 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9ht\" (UniqueName: \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.670041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.659937 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.763148 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.763235 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9ht\" (UniqueName: \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.763359 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.763528 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:24.763825 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-serving-cert: secret "maas-api-e2e-shared-a-w3-097d90-serving-cert" not found Oct 30 05:04:24.765183 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:24.763895 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls podName:ad10ddec-4609-4a69-9451-05eb371dc929 nodeName:}" failed. No retries permitted until 2026-10-30 05:04:25.263875104 +0000 UTC m=+1238.233370086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls") pod "maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" (UID: "ad10ddec-4609-4a69-9451-05eb371dc929") : secret "maas-api-e2e-shared-a-w3-097d90-serving-cert" not found Oct 30 05:04:24.766791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.766416 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.808304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.800957 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:24.936238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:24.924713 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9ht\" (UniqueName: \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:25.183651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.183596 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13" path="/var/lib/kubelet/pods/778c72c5-1e8f-4b8f-a7b3-1b97be6f6d13/volumes" Oct 30 05:04:25.187038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.186958 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" event={"ID":"976c67e7-b1c9-423c-9699-2ad23ab5667e","Type":"ContainerStarted","Data":"ba70b7432309bd30269603289ee48c8a529bff4e56dd873e15a298009213dd48"} Oct 30 05:04:25.187817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.187795 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:25.231182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.231128 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" podStartSLOduration=3.231111937 podStartE2EDuration="3.231111937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:25.229913305 +0000 UTC m=+1238.199408307" watchObservedRunningTime="2026-10-30 05:04:25.231111937 +0000 UTC m=+1238.200606925" Oct 30 05:04:25.314293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.286811 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:25.314293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.311820 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:25.592672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:25.592637 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:26.275299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.260843 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:04:26.305275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.302850 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.330700 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.330813 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.330882 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331057 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331176 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331286 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331335 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331399 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.332524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.331506 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\") pod \"8ab6873f-867f-4bb4-8712-63aec00594a7\" (UID: \"8ab6873f-867f-4bb4-8712-63aec00594a7\") " Oct 30 05:04:26.334022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.333994 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:26.336106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.334875 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:26.336274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.335473 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:26.336926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.336905 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:04:26.337271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.337209 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:26.337469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.337442 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:26.339599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.338652 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:26.339599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.339499 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "kube-api-access-7mvm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:26.341812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.341789 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:26.342085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.342062 5058 generic.go:356] "Generic (PLEG): container finished" podID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerID="4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462" exitCode=0 Oct 30 05:04:26.342275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.342112 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" event={"ID":"8ab6873f-867f-4bb4-8712-63aec00594a7","Type":"ContainerDied","Data":"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462"} Oct 30 05:04:26.342357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.342318 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" event={"ID":"8ab6873f-867f-4bb4-8712-63aec00594a7","Type":"ContainerDied","Data":"b697ee5762de208b3027fd5937f86d460fa010f17058df7a0b6363ef176284f0"} Oct 30 05:04:26.342429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.342359 5058 scope.go:122] "RemoveContainer" containerID="4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462" Oct 30 05:04:26.342484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.342150 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx" Oct 30 05:04:26.361772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.358876 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo" pod "8ab6873f-867f-4bb4-8712-63aec00594a7" (UID: "8ab6873f-867f-4bb4-8712-63aec00594a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448377 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448409 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448423 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448436 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvm4\" (UniqueName: \"kubernetes.io/projected/8ab6873f-867f-4bb4-8712-63aec00594a7-kube-api-access-7mvm4\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448452 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448574 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448588 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448601 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6873f-867f-4bb4-8712-63aec00594a7-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.505258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.448615 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6873f-867f-4bb4-8712-63aec00594a7-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:26.513784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.513424 5058 scope.go:122] "RemoveContainer" containerID="4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462" Oct 30 05:04:26.513935 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:26.513818 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462\": container with ID starting with 4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462 not found: ID does not exist" containerID="4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462" Oct 30 05:04:26.513935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.513851 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462"} err="failed to get container status \"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462\": rpc error: code = NotFound desc = could not find container \"4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462\": container with ID starting with 4cd9f3a9e5ea63dfc5004e026dc4934e7dc8e9b2055361f9359bee35e8e02462 not found: ID does not exist" Oct 30 05:04:26.718733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.718690 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:04:26.731275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:26.731248 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx"] Oct 30 05:04:27.222518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.222438 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" path="/var/lib/kubelet/pods/8ab6873f-867f-4bb4-8712-63aec00594a7/volumes" Oct 30 05:04:27.405493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.405449 5058 generic.go:356] "Generic (PLEG): container finished" podID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerID="43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5" exitCode=0 Oct 30 05:04:27.405896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.405590 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerDied","Data":"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5"} Oct 30 05:04:27.406088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.406070 5058 scope.go:122] "RemoveContainer" containerID="43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5" Oct 30 05:04:27.408094 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.408056 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.408277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.408152 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.408277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.408203 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.409688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.409663 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" event={"ID":"ad10ddec-4609-4a69-9451-05eb371dc929","Type":"ContainerStarted","Data":"c3f6f04435f75cd915de0d530941fc1e46ab953c3b87623a4029311944caa706"} Oct 30 05:04:27.409784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.409720 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" event={"ID":"ad10ddec-4609-4a69-9451-05eb371dc929","Type":"ContainerStarted","Data":"16adcdbc92f35c7508d2267ca4dce754c3ced2ca7af150cefbbf1030542724b9"} Oct 30 05:04:27.410448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:27.410421 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:28.371593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:28.371563 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:04:28.428952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:28.428914 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerStarted","Data":"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f"} Oct 30 05:04:28.513240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:28.509702 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" podStartSLOduration=4.148154581 podStartE2EDuration="4.509681703s" totalImagesPullingTime="361.527122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:27.514567975 +0000 UTC m=+1240.484062965" watchObservedRunningTime="2026-10-30 05:04:28.509681703 +0000 UTC m=+1241.479176694" Oct 30 05:04:29.310244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:29.310103 5058 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:04:29.310244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:29.310165 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:04:29.409441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:29.396932 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:04:30.311716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:30.311677 5058 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:04:30.312342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:30.312308 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:04:31.191279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:31.189709 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:04:31.352881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:31.352842 5058 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:04:31.395247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:31.394873 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:04:31.646302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:31.636903 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" containerID="cri-o://ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f" gracePeriod=30 Oct 30 05:04:33.454559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:33.454525 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:04:37.402371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.402334 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:04:37.403053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.403038 5058 cpu_manager.go:414] "containerMap: removing container" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerName="istio-proxy" Oct 30 05:04:37.403096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.403054 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerName="istio-proxy" Oct 30 05:04:37.403226 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.403199 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab6873f-867f-4bb4-8712-63aec00594a7" containerName="istio-proxy" Oct 30 05:04:37.406991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.406975 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.413920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.413897 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:04:37.425570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.425538 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:04:37.432500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.432478 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-47b1fc31\"" Oct 30 05:04:37.456858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.456826 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-47b1fc31-dockercfg-r57kb\"" Oct 30 05:04:37.529352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.505274 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.529352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.505348 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zmm\" (UniqueName: \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.616242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.609578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.616242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.609656 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zmm\" (UniqueName: \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.616242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.611121 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.859305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.858979 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zmm\" (UniqueName: \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\") pod \"payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:37.920278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:37.915934 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:38.287039 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:38.287000 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bd22c9_986a_46a0_b6a2_8e5bc75ed0a1.slice/crio-e371f77dfb8e353f822ef048bfa6ead5a6a86f803e264644d69434dcb5b96cea WatchSource:0}: Error finding container e371f77dfb8e353f822ef048bfa6ead5a6a86f803e264644d69434dcb5b96cea: Status 404 returned error can't find the container with id e371f77dfb8e353f822ef048bfa6ead5a6a86f803e264644d69434dcb5b96cea Oct 30 05:04:38.290244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.290201 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:04:38.432985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.432150 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:04:38.471734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.471699 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509093 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509170 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509310 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509343 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509436 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklv9\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509545 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509577 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509616 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.511359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.509724 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.526297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.525352 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:04:38.611731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611647 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.611731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611708 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611765 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611863 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611896 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.611999 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklv9\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612112 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612124 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612148 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612184 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612188 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612390 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612450 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.612824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.612805 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.614139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.614118 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.614382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.614366 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.632982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.632958 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.633101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.633075 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklv9\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.814599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.814562 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:38.923426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.923386 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" event={"ID":"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1","Type":"ContainerStarted","Data":"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811"} Oct 30 05:04:38.923586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.923459 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" event={"ID":"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1","Type":"ContainerStarted","Data":"e371f77dfb8e353f822ef048bfa6ead5a6a86f803e264644d69434dcb5b96cea"} Oct 30 05:04:38.923732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.923707 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:38.970098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:38.970029 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" podStartSLOduration=1.970009167 podStartE2EDuration="1.970009167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:38.966474378 +0000 UTC m=+1251.935969369" watchObservedRunningTime="2026-10-30 05:04:38.970009167 +0000 UTC m=+1251.939504158" Oct 30 05:04:39.230555 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:39.229654 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc22070_efee_4345_8efe_ab9646336281.slice/crio-20f4babe7801a42ce1191069fa87a019c1c215a24e492aa84de4a2e9457d8acb WatchSource:0}: Error finding container 20f4babe7801a42ce1191069fa87a019c1c215a24e492aa84de4a2e9457d8acb: Status 404 returned error can't find the container with id 20f4babe7801a42ce1191069fa87a019c1c215a24e492aa84de4a2e9457d8acb Oct 30 05:04:39.280625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.280588 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:04:39.285277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.285195 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:39.285461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.285310 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:39.285461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.285352 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:39.933802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.933764 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" event={"ID":"9dc22070-efee-4345-8efe-ab9646336281","Type":"ContainerStarted","Data":"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f"} Oct 30 05:04:39.934341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:39.933822 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" event={"ID":"9dc22070-efee-4345-8efe-ab9646336281","Type":"ContainerStarted","Data":"20f4babe7801a42ce1191069fa87a019c1c215a24e492aa84de4a2e9457d8acb"} Oct 30 05:04:40.006730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.006677 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podStartSLOduration=2.00666198 podStartE2EDuration="2.00666198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:39.989644253 +0000 UTC m=+1252.959139242" watchObservedRunningTime="2026-10-30 05:04:40.00666198 +0000 UTC m=+1252.976156969" Oct 30 05:04:40.561816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.561783 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:04:40.591145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591106 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591149 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591189 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591245 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591314 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591386 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591474 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591501 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591535 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.591834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591779 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\") pod \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\" (UID: \"9b083858-3553-4ad6-9f5e-eedec2a7bb24\") " Oct 30 05:04:40.591834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591812 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.591942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.591831 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.592244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592078 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.592449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592418 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:40.592619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592600 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.592689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592625 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.592689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592639 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.592689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592655 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.592689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.592671 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.593804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.593779 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "kube-api-access-74vgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:40.593906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.593881 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.594169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.593973 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:04:40.594169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.593994 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token" pod "9b083858-3553-4ad6-9f5e-eedec2a7bb24" (UID: "9b083858-3553-4ad6-9f5e-eedec2a7bb24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:40.694040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.694002 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.694040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.694034 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vgd\" (UniqueName: \"kubernetes.io/projected/9b083858-3553-4ad6-9f5e-eedec2a7bb24-kube-api-access-74vgd\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.694040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.694046 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.694307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.694055 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b083858-3553-4ad6-9f5e-eedec2a7bb24-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.815139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.815058 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:40.816777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.816750 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:40.816896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.816801 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:40.956824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.956788 5058 generic.go:356] "Generic (PLEG): container finished" podID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerID="ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f" exitCode=0 Oct 30 05:04:40.957309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.956859 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" Oct 30 05:04:40.957309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.956875 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerDied","Data":"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f"} Oct 30 05:04:40.957309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.956936 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr" event={"ID":"9b083858-3553-4ad6-9f5e-eedec2a7bb24","Type":"ContainerDied","Data":"80b4743171f2b6e08f0116f780c756c9a5e6cca585014f7b92384ddcf5b71f64"} Oct 30 05:04:40.957309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.956969 5058 scope.go:122] "RemoveContainer" containerID="ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f" Oct 30 05:04:40.969793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.969774 5058 scope.go:122] "RemoveContainer" containerID="43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5" Oct 30 05:04:40.979920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.979897 5058 scope.go:122] "RemoveContainer" containerID="ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f" Oct 30 05:04:40.980184 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:40.980162 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f\": container with ID starting with ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f not found: ID does not exist" containerID="ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f" Oct 30 05:04:40.980309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.980197 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f"} err="failed to get container status \"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f\": rpc error: code = NotFound desc = could not find container \"ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f\": container with ID starting with ef4eadaf8d3afaefaae77cbea5d8a3cff4a8156cfe311fce00c2bf9dcb1cca0f not found: ID does not exist" Oct 30 05:04:40.980309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.980257 5058 scope.go:122] "RemoveContainer" containerID="43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5" Oct 30 05:04:40.980534 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:40.980515 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5\": container with ID starting with 43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5 not found: ID does not exist" containerID="43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5" Oct 30 05:04:40.980579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:40.980540 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5"} err="failed to get container status \"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5\": rpc error: code = NotFound desc = could not find container \"43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5\": container with ID starting with 43d6ea6c7d290523962024ce08fa39ef368c67589f98c4d582a7cd28763aa6a5 not found: ID does not exist" Oct 30 05:04:41.017523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.017490 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:04:41.021507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.021479 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr"] Oct 30 05:04:41.164353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.164319 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" path="/var/lib/kubelet/pods/9b083858-3553-4ad6-9f5e-eedec2a7bb24/volumes" Oct 30 05:04:41.784008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.783977 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:04:41.784750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.784735 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:41.784806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.784751 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:41.784806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.784769 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:41.784806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.784775 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:41.784904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.784891 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:41.789824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.789805 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:41.809053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.809022 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:04:41.819338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.819305 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:41.819513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.819357 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:41.819716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.819694 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5a926a9d-serving-cert\"" Oct 30 05:04:41.912786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.912747 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:41.912999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.912809 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brfn\" (UniqueName: \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:41.913292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.913213 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:41.913357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:41.913311 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.014746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.014712 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.014746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.014752 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.015468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.014804 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.015468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.014858 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brfn\" (UniqueName: \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.015688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.015665 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.017375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.017354 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.017981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.017963 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.050990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.050908 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brfn\" (UniqueName: \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\") pod \"maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.111246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.107558 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:42.620742 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:04:42.620705 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f914e5e_4a75_43ea_99e2_c8d8ac99f95a.slice/crio-022ed61f97f46d0b753c93519eec2114b9299386bc35a24ac6b55eedf89e62fa WatchSource:0}: Error finding container 022ed61f97f46d0b753c93519eec2114b9299386bc35a24ac6b55eedf89e62fa: Status 404 returned error can't find the container with id 022ed61f97f46d0b753c93519eec2114b9299386bc35a24ac6b55eedf89e62fa Oct 30 05:04:42.623236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.623184 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:04:42.815833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.815798 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:42.816023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.815858 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:42.974626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:42.974539 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" event={"ID":"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a","Type":"ContainerStarted","Data":"022ed61f97f46d0b753c93519eec2114b9299386bc35a24ac6b55eedf89e62fa"} Oct 30 05:04:43.848244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:43.844593 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:43.848244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:43.844664 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:44.224254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:44.191779 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" event={"ID":"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a","Type":"ContainerStarted","Data":"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c"} Oct 30 05:04:44.224254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:44.191849 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:44.282242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:44.281629 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" podStartSLOduration=2.6264287790000003 podStartE2EDuration="3.28160657s" totalImagesPullingTime="655.177791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:44.239518093 +0000 UTC m=+1257.209013084" watchObservedRunningTime="2026-10-30 05:04:44.28160657 +0000 UTC m=+1257.251101561" Oct 30 05:04:44.815349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:44.815211 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:44.815349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:44.815295 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:45.820242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:45.817622 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:45.820242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:45.817684 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:46.820243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:46.814937 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:46.820243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:46.815003 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:47.884658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:47.884475 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:47.893430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:47.893393 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:04:47.893626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:47.893543 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 05:04:49.306612 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:49.306559 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:04:49.862430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:49.862388 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:04:49.862627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:49.862470 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:49.950436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:49.950400 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:04:50.285805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:50.285650 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:04:50.694834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:50.694799 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:50.696179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:50.696155 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:50.696604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:50.696584 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:04:57.098412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.098335 5058 generic.go:356] "Generic (PLEG): container finished" podID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerID="eade0cae3c9157218e8898a5af77bb04f01f00e0937ec2ea3285db199c5aaddd" exitCode=0 Oct 30 05:04:57.098802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.098415 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerDied","Data":"eade0cae3c9157218e8898a5af77bb04f01f00e0937ec2ea3285db199c5aaddd"} Oct 30 05:04:57.098937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.098921 5058 scope.go:122] "RemoveContainer" containerID="eade0cae3c9157218e8898a5af77bb04f01f00e0937ec2ea3285db199c5aaddd" Oct 30 05:04:57.100686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.100658 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:57.100767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.100729 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:57.100767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.100760 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:57.606804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.606766 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:04:57.613182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.613152 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b083858-3553-4ad6-9f5e-eedec2a7bb24" containerName="istio-proxy" Oct 30 05:04:57.616697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.616632 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.650241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.641128 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:04:57.653251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.651499 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-47b1fc31-serving-cert\"" Oct 30 05:04:57.757523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.757478 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.757706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.757535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjct\" (UniqueName: \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.757706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.757610 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.758076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.758041 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.866026 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.866156 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.866185 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjct\" (UniqueName: \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.866277 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:57.866660 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-serving-cert: secret "maas-api-e2e-derive-47b1fc31-serving-cert" not found Oct 30 05:04:57.869422 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:04:57.866737 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls podName:b465e73f-e614-4fec-a847-715a10eccebb nodeName:}" failed. No retries permitted until 2026-10-30 05:04:58.366713187 +0000 UTC m=+1271.336208177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls") pod "maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" (UID: "b465e73f-e614-4fec-a847-715a10eccebb") : secret "maas-api-e2e-derive-47b1fc31-serving-cert" not found Oct 30 05:04:57.932207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.900710 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.933270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.933189 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:57.962269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:57.951817 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjct\" (UniqueName: \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:58.115556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.115518 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerStarted","Data":"49439db0b272e97d53f9dd7cfb86fab39eb364940daff4aff691cae9454ef08f"} Oct 30 05:04:58.268613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.268587 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:04:58.277754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.277726 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:04:58.278909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.278885 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:04:58.278984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.278932 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:04:58.401550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.401516 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:58.403938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.403918 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") pod \"maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:58.585293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:58.580540 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:04:59.004637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.003792 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:04:59.011105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.011067 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.017799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.016628 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:04:59.052326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.042880 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:04:59.052326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.045461 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e15c99-dockercfg-5ftfk\"" Oct 30 05:04:59.052326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.045559 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e15c99\"" Oct 30 05:04:59.159241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.158743 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.159241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.158903 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49smf\" (UniqueName: \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.295290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.262689 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.337242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.333424 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49smf\" (UniqueName: \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.348268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.277541 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:04:59.348268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.344667 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:04:59.368447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.285949 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.439948 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.439899 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" event={"ID":"b465e73f-e614-4fec-a847-715a10eccebb","Type":"ContainerStarted","Data":"161747bbebadb67a31657d2e82ae21ed32cd1f8be273f49b50444c5e54950459"} Oct 30 05:04:59.489270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.486022 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49smf\" (UniqueName: \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\") pod \"payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:04:59.686409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:04:59.684515 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:05:00.156718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.148035 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:05:00.173907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.173875 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" event={"ID":"af53fc3a-8d64-482e-9afd-93e12cebe055","Type":"ContainerStarted","Data":"aa16e47f14557444ad6a409f2905a0c52a59d4a1927af4e61f8c7b912c025331"} Oct 30 05:05:00.175959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.175935 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" event={"ID":"b465e73f-e614-4fec-a847-715a10eccebb","Type":"ContainerStarted","Data":"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001"} Oct 30 05:05:00.176247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.176189 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:05:00.278161 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.278075 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:05:00.278161 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:00.278123 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:05:01.184680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.184644 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" event={"ID":"af53fc3a-8d64-482e-9afd-93e12cebe055","Type":"ContainerStarted","Data":"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10"} Oct 30 05:05:01.185149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.184731 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:05:01.250465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.250407 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" podStartSLOduration=3.250384388 podStartE2EDuration="3.250384388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:01.247661124 +0000 UTC m=+1274.217156114" watchObservedRunningTime="2026-10-30 05:05:01.250384388 +0000 UTC m=+1274.219879377" Oct 30 05:05:01.252532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.252481 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" podStartSLOduration=3.661933076 podStartE2EDuration="4.252469941s" totalImagesPullingTime="590.536865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:00.229879211 +0000 UTC m=+1273.199374202" watchObservedRunningTime="2026-10-30 05:05:01.252469941 +0000 UTC m=+1274.221964978" Oct 30 05:05:01.277533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.277502 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:05:01.277699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:01.277555 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:05:02.308910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:02.284698 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:05:02.308910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:02.284771 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:05:03.278150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:03.278116 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:05:03.278367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:03.278178 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:05:04.281988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:04.281960 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:05:04.283173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:04.283148 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:05:06.190467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:06.190440 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:05:12.286254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:12.286207 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:05:18.373823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.373789 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:05:18.381198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.381163 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.405095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.405063 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:05:18.426144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.426109 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e15c99-serving-cert\"" Oct 30 05:05:18.529859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.529821 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.530071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.530013 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrh59\" (UniqueName: \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.530166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.530149 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.530241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.530198 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.633662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.633574 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.641250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.634614 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh59\" (UniqueName: \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.641719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.641688 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.688493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.688451 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.691566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.671043 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh59\" (UniqueName: \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.696359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.643480 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.696555 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.694480 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.700918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.700888 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:18.704785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:18.704750 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:19.618489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:19.618449 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:05:19.621032 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:05:19.620985 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:05:20.590320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:20.590280 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" event={"ID":"569e14f1-b49c-4832-80d9-f7c5a2c48fad","Type":"ContainerStarted","Data":"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1"} Oct 30 05:05:20.590517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:20.590340 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" event={"ID":"569e14f1-b49c-4832-80d9-f7c5a2c48fad","Type":"ContainerStarted","Data":"9ac7af78a73b15ec6ca5a654aeb80b7ba83ee797207ce1b6493c7e5617c9fc46"} Oct 30 05:05:20.590517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:20.590414 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:20.639036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:20.638978 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" podStartSLOduration=2.2915552630000002 podStartE2EDuration="2.638963342s" totalImagesPullingTime="347.408079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:20.634980172 +0000 UTC m=+1293.604475173" watchObservedRunningTime="2026-10-30 05:05:20.638963342 +0000 UTC m=+1293.608458338" Oct 30 05:05:23.520127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:23.520092 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:05:26.604016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:26.603981 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:05:28.520901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.520773 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" podUID="976c67e7-b1c9-423c-9699-2ad23ab5667e" containerName="payload-processing" containerID="cri-o://ba70b7432309bd30269603289ee48c8a529bff4e56dd873e15a298009213dd48" gracePeriod=25 Oct 30 05:05:28.798822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.798790 5058 generic.go:356] "Generic (PLEG): container finished" podID="976c67e7-b1c9-423c-9699-2ad23ab5667e" containerID="ba70b7432309bd30269603289ee48c8a529bff4e56dd873e15a298009213dd48" exitCode=0 Oct 30 05:05:28.798973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.798856 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" event={"ID":"976c67e7-b1c9-423c-9699-2ad23ab5667e","Type":"ContainerDied","Data":"ba70b7432309bd30269603289ee48c8a529bff4e56dd873e15a298009213dd48"} Oct 30 05:05:28.798973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.798908 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" event={"ID":"976c67e7-b1c9-423c-9699-2ad23ab5667e","Type":"ContainerDied","Data":"070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8"} Oct 30 05:05:28.798973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.798927 5058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="070b12f2e48f4c07a3f50b51074983fbc0f9638454eb831738f1dcaafca8d7d8" Oct 30 05:05:28.818583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.818560 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:05:28.900359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.900322 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\" (UniqueName: \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\") pod \"976c67e7-b1c9-423c-9699-2ad23ab5667e\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " Oct 30 05:05:28.900601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.900581 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\") pod \"976c67e7-b1c9-423c-9699-2ad23ab5667e\" (UID: \"976c67e7-b1c9-423c-9699-2ad23ab5667e\") " Oct 30 05:05:28.901085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.901038 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume" pod "976c67e7-b1c9-423c-9699-2ad23ab5667e" (UID: "976c67e7-b1c9-423c-9699-2ad23ab5667e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:28.901518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.901493 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c67e7-b1c9-423c-9699-2ad23ab5667e-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:28.903079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:28.903051 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4" pod "976c67e7-b1c9-423c-9699-2ad23ab5667e" (UID: "976c67e7-b1c9-423c-9699-2ad23ab5667e"). InnerVolumeSpecName "kube-api-access-dmdm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:29.003736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:29.003694 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdm4\" (UniqueName: \"kubernetes.io/projected/976c67e7-b1c9-423c-9699-2ad23ab5667e-kube-api-access-dmdm4\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:29.868281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:29.859685 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q" Oct 30 05:05:29.957053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:29.957017 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:05:29.963446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:29.963345 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q"] Oct 30 05:05:31.165843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.165803 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976c67e7-b1c9-423c-9699-2ad23ab5667e" path="/var/lib/kubelet/pods/976c67e7-b1c9-423c-9699-2ad23ab5667e/volumes" Oct 30 05:05:31.304672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.304638 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:05:31.304908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.304887 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" podUID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" containerName="maas-api" containerID="cri-o://57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c" gracePeriod=30 Oct 30 05:05:31.817679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.817638 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:05:31.859142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.859115 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\") pod \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " Oct 30 05:05:31.859450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.859435 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\") pod \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " Oct 30 05:05:31.859884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.859581 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca" pod "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" (UID: "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:31.860175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.860145 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\" (UniqueName: \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\") pod \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " Oct 30 05:05:31.860510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.860494 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\") pod \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\" (UID: \"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a\") " Oct 30 05:05:31.861374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.861353 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:31.862598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.862579 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs" pod "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" (UID: "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:31.862920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.862894 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn" pod "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" (UID: "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a"). InnerVolumeSpecName "kube-api-access-9brfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:31.863642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.863615 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls" pod "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" (UID: "2f914e5e-4a75-43ea-99e2-c8d8ac99f95a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:31.880242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.877821 5058 generic.go:356] "Generic (PLEG): container finished" podID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" containerID="57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c" exitCode=0 Oct 30 05:05:31.880242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.877965 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" event={"ID":"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a","Type":"ContainerDied","Data":"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c"} Oct 30 05:05:31.880242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.878015 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" event={"ID":"2f914e5e-4a75-43ea-99e2-c8d8ac99f95a","Type":"ContainerDied","Data":"022ed61f97f46d0b753c93519eec2114b9299386bc35a24ac6b55eedf89e62fa"} Oct 30 05:05:31.880242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.878044 5058 scope.go:122] "RemoveContainer" containerID="57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c" Oct 30 05:05:31.880242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.878261 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv" Oct 30 05:05:31.906002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.905974 5058 scope.go:122] "RemoveContainer" containerID="57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c" Oct 30 05:05:31.906368 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:05:31.906342 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c\": container with ID starting with 57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c not found: ID does not exist" containerID="57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c" Oct 30 05:05:31.906483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.906376 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c"} err="failed to get container status \"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c\": rpc error: code = NotFound desc = could not find container \"57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c\": container with ID starting with 57081398863b6cee58e47db17077688678b7ebcd449ce6edca6c337ff6e5329c not found: ID does not exist" Oct 30 05:05:31.947318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.947287 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:05:31.951379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.951345 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a926a9d-65c59468b8-mnkdv"] Oct 30 05:05:31.963324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.963288 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:31.963324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.963315 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brfn\" (UniqueName: \"kubernetes.io/projected/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-kube-api-access-9brfn\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:31.963324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:31.963325 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:33.165616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.165574 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" path="/var/lib/kubelet/pods/2f914e5e-4a75-43ea-99e2-c8d8ac99f95a/volumes" Oct 30 05:05:33.785865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.785811 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:05:33.787051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787026 5058 cpu_manager.go:414] "containerMap: removing container" podUID="976c67e7-b1c9-423c-9699-2ad23ab5667e" containerName="payload-processing" Oct 30 05:05:33.787163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787052 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976c67e7-b1c9-423c-9699-2ad23ab5667e" containerName="payload-processing" Oct 30 05:05:33.787163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787068 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" containerName="maas-api" Oct 30 05:05:33.787163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787076 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" containerName="maas-api" Oct 30 05:05:33.787357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787336 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="976c67e7-b1c9-423c-9699-2ad23ab5667e" containerName="payload-processing" Oct 30 05:05:33.787357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.787351 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f914e5e-4a75-43ea-99e2-c8d8ac99f95a" containerName="maas-api" Oct 30 05:05:33.793198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.793177 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:33.822837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.822802 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:05:33.828387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.828361 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-44bc69-openshift-default-dockercfg-nfx8p\"" Oct 30 05:05:34.049298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:33.993858 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.143506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.143454 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.143829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.143806 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.143994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.143974 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.144127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.144111 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.144324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.144309 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.144462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.144448 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.144645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.144631 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4sk7\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.144789 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.144776 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248391 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248592 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248638 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248673 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248751 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248807 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248905 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4sk7\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.248972 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.249023 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.250101 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.256246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.250785 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.257755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.257722 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.258205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.258181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.259635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.258661 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.297242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.295759 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.300387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.299901 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4sk7\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.301133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.300531 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.301133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.301016 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.411679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.411643 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:34.813588 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:05:34.813555 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049cc2ae_2247_43f9_8b0c_7ffba3fe7170.slice/crio-dac6879dc4c7e583a1db405ac2d40b09bd99d5d1aaeee39e89c0128c55da0302 WatchSource:0}: Error finding container dac6879dc4c7e583a1db405ac2d40b09bd99d5d1aaeee39e89c0128c55da0302: Status 404 returned error can't find the container with id dac6879dc4c7e583a1db405ac2d40b09bd99d5d1aaeee39e89c0128c55da0302 Oct 30 05:05:34.815427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.815334 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:05:34.816407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.816376 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:34.816483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.816446 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:34.816526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:34.816483 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:35.168460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.168427 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerStarted","Data":"e5e341063ceb9880064af3f2334fe534768fdf90d16f13619f3886ac02cbd8f9"} Oct 30 05:05:35.168662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.168475 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerStarted","Data":"dac6879dc4c7e583a1db405ac2d40b09bd99d5d1aaeee39e89c0128c55da0302"} Oct 30 05:05:35.243031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.242965 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podStartSLOduration=2.242943702 podStartE2EDuration="2.242943702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:35.228178995 +0000 UTC m=+1308.197673986" watchObservedRunningTime="2026-10-30 05:05:35.242943702 +0000 UTC m=+1308.212438693" Oct 30 05:05:35.437292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.427482 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:35.437292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.430401 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:05:35.437292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:35.430464 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:05:36.418184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:36.418156 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:36.916865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:36.916829 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:05:37.268586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:37.268488 5058 generic.go:356] "Generic (PLEG): container finished" podID="dbd5ac21-64cd-46ae-bea8-6047761a685d" containerID="93bee2a205abd3f56b03a04fe46584e2cde624c27e5846a38d86d7c95b471929" exitCode=0 Oct 30 05:05:37.270109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:37.270079 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" event={"ID":"dbd5ac21-64cd-46ae-bea8-6047761a685d","Type":"ContainerDied","Data":"93bee2a205abd3f56b03a04fe46584e2cde624c27e5846a38d86d7c95b471929"} Oct 30 05:05:37.270312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:37.270137 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:37.271875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:37.271835 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:05:38.908304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:38.908273 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:05:39.012307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.010989 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\") pod \"dbd5ac21-64cd-46ae-bea8-6047761a685d\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " Oct 30 05:05:39.012307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.011061 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\" (UniqueName: \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\") pod \"dbd5ac21-64cd-46ae-bea8-6047761a685d\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " Oct 30 05:05:39.012307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.011206 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\") pod \"dbd5ac21-64cd-46ae-bea8-6047761a685d\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " Oct 30 05:05:39.012307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.011450 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\") pod \"dbd5ac21-64cd-46ae-bea8-6047761a685d\" (UID: \"dbd5ac21-64cd-46ae-bea8-6047761a685d\") " Oct 30 05:05:39.012993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.012970 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle" pod "dbd5ac21-64cd-46ae-bea8-6047761a685d" (UID: "dbd5ac21-64cd-46ae-bea8-6047761a685d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:39.013448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.013422 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path" pod "dbd5ac21-64cd-46ae-bea8-6047761a685d" (UID: "dbd5ac21-64cd-46ae-bea8-6047761a685d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:39.014233 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.014196 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle" pod "dbd5ac21-64cd-46ae-bea8-6047761a685d" (UID: "dbd5ac21-64cd-46ae-bea8-6047761a685d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:39.020641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.020623 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5" pod "dbd5ac21-64cd-46ae-bea8-6047761a685d" (UID: "dbd5ac21-64cd-46ae-bea8-6047761a685d"). InnerVolumeSpecName "kube-api-access-4b5h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:39.114768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.114732 5058 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-trusted-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:39.115012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.114990 5058 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbd5ac21-64cd-46ae-bea8-6047761a685d-archives-path\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:39.115142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.115128 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b5h5\" (UniqueName: \"kubernetes.io/projected/dbd5ac21-64cd-46ae-bea8-6047761a685d-kube-api-access-4b5h5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:39.115253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.115240 5058 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbd5ac21-64cd-46ae-bea8-6047761a685d-service-ca-bundle\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:39.412759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.412669 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" Oct 30 05:05:39.412759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.412705 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ns7rz-vvjmg" event={"ID":"dbd5ac21-64cd-46ae-bea8-6047761a685d","Type":"ContainerDied","Data":"24b3280b10f89359ef38ddd552a39212c1aea169f824553b2e2b96a84dbbeac3"} Oct 30 05:05:39.412759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:39.412761 5058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b3280b10f89359ef38ddd552a39212c1aea169f824553b2e2b96a84dbbeac3" Oct 30 05:05:41.918136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:41.918096 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" podUID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" containerName="payload-processing" containerID="cri-o://81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811" gracePeriod=25 Oct 30 05:05:42.209033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.209007 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:05:42.218735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.218696 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\" (UniqueName: \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\") pod \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " Oct 30 05:05:42.218896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.218820 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\") pod \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\" (UID: \"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1\") " Oct 30 05:05:42.219300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.219267 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume" pod "b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" (UID: "b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:42.219570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.219551 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:42.220805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.220785 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm" pod "b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" (UID: "b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1"). InnerVolumeSpecName "kube-api-access-s2zmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:42.323847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.323797 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2zmm\" (UniqueName: \"kubernetes.io/projected/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1-kube-api-access-s2zmm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:42.454203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.454110 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" Oct 30 05:05:42.456654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.454826 5058 generic.go:356] "Generic (PLEG): container finished" podID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" containerID="81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811" exitCode=0 Oct 30 05:05:42.456654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.454856 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" event={"ID":"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1","Type":"ContainerDied","Data":"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811"} Oct 30 05:05:42.456654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.455951 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d" event={"ID":"b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1","Type":"ContainerDied","Data":"e371f77dfb8e353f822ef048bfa6ead5a6a86f803e264644d69434dcb5b96cea"} Oct 30 05:05:42.456654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.455987 5058 scope.go:122] "RemoveContainer" containerID="81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811" Oct 30 05:05:42.473646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.473617 5058 scope.go:122] "RemoveContainer" containerID="81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811" Oct 30 05:05:42.473996 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:05:42.473974 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811\": container with ID starting with 81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811 not found: ID does not exist" containerID="81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811" Oct 30 05:05:42.474083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.474011 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811"} err="failed to get container status \"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811\": rpc error: code = NotFound desc = could not find container \"81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811\": container with ID starting with 81e09487ef0f74872abf48eaafb8ad70b03e0cecfa24a9fd24dc9241e93c9811 not found: ID does not exist" Oct 30 05:05:42.513746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.513709 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:05:42.519552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:42.519516 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d"] Oct 30 05:05:43.165435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:43.165395 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" path="/var/lib/kubelet/pods/b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1/volumes" Oct 30 05:05:48.705980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.705941 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:05:48.706986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.706955 5058 cpu_manager.go:414] "containerMap: removing container" podUID="dbd5ac21-64cd-46ae-bea8-6047761a685d" containerName="insights-gathering" Oct 30 05:05:48.706986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.706981 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd5ac21-64cd-46ae-bea8-6047761a685d" containerName="insights-gathering" Oct 30 05:05:48.707168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.707023 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" containerName="payload-processing" Oct 30 05:05:48.707168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.707032 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" containerName="payload-processing" Oct 30 05:05:48.707300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.707232 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd5ac21-64cd-46ae-bea8-6047761a685d" containerName="insights-gathering" Oct 30 05:05:48.707300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.707257 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bd22c9-986a-46a0-b6a2-8e5bc75ed0a1" containerName="payload-processing" Oct 30 05:05:48.728311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.728274 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.749417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.749382 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:05:48.772745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.772715 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c613f5ea-serving-cert\"" Oct 30 05:05:48.868998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.868950 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.869168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.869140 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.869238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.869210 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fqn\" (UniqueName: \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.869333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.869313 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.976292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.975509 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.976292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.975624 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fqn\" (UniqueName: \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.976292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.975672 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.976292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.975917 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:48.983241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:48.981077 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:49.008264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.000421 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:49.009813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.008898 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:49.037237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.026714 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fqn\" (UniqueName: \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:49.073244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.072567 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:49.215730 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:05:49.215588 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:05:49.458720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.458690 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:05:49.520891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:49.520854 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" event={"ID":"540aa781-d6af-4ad1-827b-c09551d77cab","Type":"ContainerStarted","Data":"ce989a9e66355f92a7c45a61e67ce1ebcdaa6dc1c2ddebbe60db5ad918807786"} Oct 30 05:05:50.791806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:50.791760 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" event={"ID":"540aa781-d6af-4ad1-827b-c09551d77cab","Type":"ContainerStarted","Data":"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e"} Oct 30 05:05:50.792362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:50.792072 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:05:50.856335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:50.856264 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" podStartSLOduration=2.457607684 podStartE2EDuration="2.856247131s" totalImagesPullingTime="398.639447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:50.852098381 +0000 UTC m=+1323.821593371" watchObservedRunningTime="2026-10-30 05:05:50.856247131 +0000 UTC m=+1323.825742120" Oct 30 05:05:51.055571 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:51.055485 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:05:51.056059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:51.056028 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" podUID="b465e73f-e614-4fec-a847-715a10eccebb" containerName="maas-api" containerID="cri-o://7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001" gracePeriod=30 Oct 30 05:05:51.326108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:51.324798 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" podUID="b465e73f-e614-4fec-a847-715a10eccebb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.103:8443/health\": dial tcp 10.132.0.103:8443: connect: connection refused" Oct 30 05:05:52.138333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.137976 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:05:52.146638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.146603 5058 generic.go:356] "Generic (PLEG): container finished" podID="b465e73f-e614-4fec-a847-715a10eccebb" containerID="7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001" exitCode=0 Oct 30 05:05:52.146806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.146661 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" Oct 30 05:05:52.146806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.146673 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" event={"ID":"b465e73f-e614-4fec-a847-715a10eccebb","Type":"ContainerDied","Data":"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001"} Oct 30 05:05:52.146806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.146745 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9" event={"ID":"b465e73f-e614-4fec-a847-715a10eccebb","Type":"ContainerDied","Data":"161747bbebadb67a31657d2e82ae21ed32cd1f8be273f49b50444c5e54950459"} Oct 30 05:05:52.146806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.146771 5058 scope.go:122] "RemoveContainer" containerID="7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001" Oct 30 05:05:52.163485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.163465 5058 scope.go:122] "RemoveContainer" containerID="7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001" Oct 30 05:05:52.163909 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:05:52.163883 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001\": container with ID starting with 7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001 not found: ID does not exist" containerID="7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001" Oct 30 05:05:52.164083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.164053 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001"} err="failed to get container status \"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001\": rpc error: code = NotFound desc = could not find container \"7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001\": container with ID starting with 7c50983f87ecadac88ca834c5e61acef3babccb4f7c6039f68d52bb387263001 not found: ID does not exist" Oct 30 05:05:52.273424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.273343 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") pod \"b465e73f-e614-4fec-a847-715a10eccebb\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " Oct 30 05:05:52.273606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.273443 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\") pod \"b465e73f-e614-4fec-a847-715a10eccebb\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " Oct 30 05:05:52.273606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.273486 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\" (UniqueName: \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\") pod \"b465e73f-e614-4fec-a847-715a10eccebb\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " Oct 30 05:05:52.273729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.273656 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\") pod \"b465e73f-e614-4fec-a847-715a10eccebb\" (UID: \"b465e73f-e614-4fec-a847-715a10eccebb\") " Oct 30 05:05:52.273857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.273832 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca" pod "b465e73f-e614-4fec-a847-715a10eccebb" (UID: "b465e73f-e614-4fec-a847-715a10eccebb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:52.274528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.274506 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b465e73f-e614-4fec-a847-715a10eccebb-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:52.275700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.275673 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls" pod "b465e73f-e614-4fec-a847-715a10eccebb" (UID: "b465e73f-e614-4fec-a847-715a10eccebb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:52.275822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.275799 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs" pod "b465e73f-e614-4fec-a847-715a10eccebb" (UID: "b465e73f-e614-4fec-a847-715a10eccebb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:52.275876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.275837 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct" pod "b465e73f-e614-4fec-a847-715a10eccebb" (UID: "b465e73f-e614-4fec-a847-715a10eccebb"). InnerVolumeSpecName "kube-api-access-lzjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:52.376469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.376429 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:52.376469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.376457 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjct\" (UniqueName: \"kubernetes.io/projected/b465e73f-e614-4fec-a847-715a10eccebb-kube-api-access-lzjct\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:52.376469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.376468 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b465e73f-e614-4fec-a847-715a10eccebb-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:05:52.633254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.633159 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:05:52.637756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:52.637707 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47b1fc31-84d4c486b4-9kks9"] Oct 30 05:05:53.182597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:53.182565 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b465e73f-e614-4fec-a847-715a10eccebb" path="/var/lib/kubelet/pods/b465e73f-e614-4fec-a847-715a10eccebb/volumes" Oct 30 05:05:57.152786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:05:57.152754 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:06:00.668881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.668831 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:06:00.669703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.669686 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b465e73f-e614-4fec-a847-715a10eccebb" containerName="maas-api" Oct 30 05:06:00.669761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.669704 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b465e73f-e614-4fec-a847-715a10eccebb" containerName="maas-api" Oct 30 05:06:00.669831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.669816 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b465e73f-e614-4fec-a847-715a10eccebb" containerName="maas-api" Oct 30 05:06:00.674728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.674707 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:00.697896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.697870 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ff4760-dockercfg-75x6p\"" Oct 30 05:06:00.717785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.717744 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ff4760\"" Oct 30 05:06:00.718943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.718906 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:06:00.792245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:00.751720 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.018446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.018354 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.018992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.018829 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjj72\" (UniqueName: \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.020449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.020426 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.133256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.125195 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjj72\" (UniqueName: \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.220025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.219936 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjj72\" (UniqueName: \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\") pod \"payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.290734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.290647 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:01.614544 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:06:01.614515 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06971047_7680_406d_84a7_02f31bd1a2d9.slice/crio-14d571578e88103031ed185e5bcd2b7864882c94dd236bb624df6b7d462adf54 WatchSource:0}: Error finding container 14d571578e88103031ed185e5bcd2b7864882c94dd236bb624df6b7d462adf54: Status 404 returned error can't find the container with id 14d571578e88103031ed185e5bcd2b7864882c94dd236bb624df6b7d462adf54 Oct 30 05:06:01.615630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:01.615605 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:06:02.392066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:02.392029 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" event={"ID":"06971047-7680-406d-84a7-02f31bd1a2d9","Type":"ContainerStarted","Data":"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e"} Oct 30 05:06:02.392501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:02.392092 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" event={"ID":"06971047-7680-406d-84a7-02f31bd1a2d9","Type":"ContainerStarted","Data":"14d571578e88103031ed185e5bcd2b7864882c94dd236bb624df6b7d462adf54"} Oct 30 05:06:02.392501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:02.392123 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:02.447434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:02.447379 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" podStartSLOduration=2.447365816 podStartE2EDuration="2.447365816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:02.445162443 +0000 UTC m=+1335.414657431" watchObservedRunningTime="2026-10-30 05:06:02.447365816 +0000 UTC m=+1335.416860804" Oct 30 05:06:04.165264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.163562 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:06:04.171621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.171590 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.194512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.194476 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-44bc69-serving-cert\"" Oct 30 05:06:04.199615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.199578 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:06:04.315171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.315131 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.315406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.315266 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.315534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.315514 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.315622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.315608 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jntx\" (UniqueName: \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.458109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.456444 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.458109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.456567 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.458109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.456829 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.458109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.456937 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jntx\" (UniqueName: \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.465543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.465495 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.491023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.490984 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.497875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.497845 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jntx\" (UniqueName: \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.498014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.497992 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:04.796730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:04.796645 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:05.145484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:05.145439 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:06:05.482155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:05.482068 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" event={"ID":"e4857400-6961-4a25-9057-a70946e58b9b","Type":"ContainerStarted","Data":"ac1ad89f0d357464c9d25a209e4ca088ac1f80dc294519332929cb7c940f2229"} Oct 30 05:06:06.545702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:06.545656 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" event={"ID":"e4857400-6961-4a25-9057-a70946e58b9b","Type":"ContainerStarted","Data":"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632"} Oct 30 05:06:06.547618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:06.547593 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:06.597880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:06.597822 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" podStartSLOduration=2.251591742 podStartE2EDuration="2.597805948s" totalImagesPullingTime="346.214206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:06.593084553 +0000 UTC m=+1339.562579544" watchObservedRunningTime="2026-10-30 05:06:06.597805948 +0000 UTC m=+1339.567300970" Oct 30 05:06:13.402440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:13.402412 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:06:13.567093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:13.567069 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:06:19.192280 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:06:19.192195 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:06:41.123601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.123565 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:06:41.128702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.128680 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.144935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.144907 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:06:41.153748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.153719 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-93bad9\"" Oct 30 05:06:41.153920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.153868 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-93bad9-dockercfg-xnp5l\"" Oct 30 05:06:41.199778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.199734 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfhn\" (UniqueName: \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.200035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.200010 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.360246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.316142 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfhn\" (UniqueName: \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.453595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.453557 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.493607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.454702 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.583059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.583029 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfhn\" (UniqueName: \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\") pod \"payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:41.744424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:41.744337 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:42.052444 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:06:42.052398 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60928a7_7924_46f2_a3a9_bc75a4fcc018.slice/crio-c725becf53cc49e8c6a39b740827aef8bbcd7d9f112586fb21f36a6fa31b1afb WatchSource:0}: Error finding container c725becf53cc49e8c6a39b740827aef8bbcd7d9f112586fb21f36a6fa31b1afb: Status 404 returned error can't find the container with id c725becf53cc49e8c6a39b740827aef8bbcd7d9f112586fb21f36a6fa31b1afb Oct 30 05:06:42.053877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:42.053848 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:06:42.062156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:42.062127 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" event={"ID":"f60928a7-7924-46f2-a3a9-bc75a4fcc018","Type":"ContainerStarted","Data":"c725becf53cc49e8c6a39b740827aef8bbcd7d9f112586fb21f36a6fa31b1afb"} Oct 30 05:06:43.071951 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:43.071917 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" event={"ID":"f60928a7-7924-46f2-a3a9-bc75a4fcc018","Type":"ContainerStarted","Data":"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b"} Oct 30 05:06:43.072503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:43.072149 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:43.117113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:43.117061 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" podStartSLOduration=2.117046632 podStartE2EDuration="2.117046632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:43.113898869 +0000 UTC m=+1376.083393868" watchObservedRunningTime="2026-10-30 05:06:43.117046632 +0000 UTC m=+1376.086541647" Oct 30 05:06:49.221463 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:06:49.221422 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:06:54.084257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:54.084210 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:06:55.584485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.584444 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:06:55.591915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.591887 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618303 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618387 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwx6\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618543 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618587 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618635 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618666 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618701 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618742 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.618869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.618761 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.619330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.619307 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:06:55.640871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.640842 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dfb231-openshift-default-dockercfg-t5h89\"" Oct 30 05:06:55.720296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720261 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720328 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720375 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720403 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720451 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720499 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720521 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720607 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720674 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwx6\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.720887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.720765 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.721042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.721013 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.721185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.721160 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.721286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.721184 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.721343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.721291 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.722812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.722787 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.723133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.723113 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.742764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.742730 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwx6\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.743937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.743909 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:55.920733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:55.920697 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:56.574210 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:06:56.574173 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d26a26_b1d8_4390_8883_7965d19497d4.slice/crio-5d1c573d13791040ea7fb98ca74c9d1654b20b20eaef047d3f9bd7f8e1e2ba52 WatchSource:0}: Error finding container 5d1c573d13791040ea7fb98ca74c9d1654b20b20eaef047d3f9bd7f8e1e2ba52: Status 404 returned error can't find the container with id 5d1c573d13791040ea7fb98ca74c9d1654b20b20eaef047d3f9bd7f8e1e2ba52 Oct 30 05:06:56.576256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:56.576197 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:06:56.577104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:56.577073 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:56.577181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:56.577140 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:56.577181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:56.577171 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:57.321371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:57.321334 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerStarted","Data":"4adb03779991e646532b9b1da0bfef5007433f444adc4ed5a573949b7281da85"} Oct 30 05:06:57.321886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:57.321391 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerStarted","Data":"5d1c573d13791040ea7fb98ca74c9d1654b20b20eaef047d3f9bd7f8e1e2ba52"} Oct 30 05:06:57.921700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:57.921606 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:57.928694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:57.928663 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:57.978195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:57.978134 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podStartSLOduration=2.978107579 podStartE2EDuration="2.978107579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:57.393322058 +0000 UTC m=+1390.362817044" watchObservedRunningTime="2026-10-30 05:06:57.978107579 +0000 UTC m=+1390.947602568" Oct 30 05:06:58.329209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.329129 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:58.330302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.330280 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:06:58.494320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.494286 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:06:58.730245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.701760 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:58.824597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.782911 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ef844616\"" Oct 30 05:06:58.824597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.786064 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:58.826462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.826432 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ef844616-dockercfg-ljh29\"" Oct 30 05:06:58.958048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.957843 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:06:58.964325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:58.963005 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqbt\" (UniqueName: \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.064984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.064901 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqbt\" (UniqueName: \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.065148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.065123 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.065891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.065867 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.086814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.086782 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqbt\" (UniqueName: \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\") pod \"payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.331369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.331282 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:06:59.668053 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:06:59.668021 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629bb872_a2d2_4017_9e32_8ae212d0396a.slice/crio-bff72190ea1b5cde052f6a73f20cd328cf1892a02511197cb117fa89e3115422 WatchSource:0}: Error finding container bff72190ea1b5cde052f6a73f20cd328cf1892a02511197cb117fa89e3115422: Status 404 returned error can't find the container with id bff72190ea1b5cde052f6a73f20cd328cf1892a02511197cb117fa89e3115422 Oct 30 05:06:59.669887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:06:59.669859 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:07:00.498240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:00.469558 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" event={"ID":"629bb872-a2d2-4017-9e32-8ae212d0396a","Type":"ContainerStarted","Data":"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3"} Oct 30 05:07:00.498240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:00.469628 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" event={"ID":"629bb872-a2d2-4017-9e32-8ae212d0396a","Type":"ContainerStarted","Data":"bff72190ea1b5cde052f6a73f20cd328cf1892a02511197cb117fa89e3115422"} Oct 30 05:07:00.498240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:00.469677 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:07:00.603942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:00.557914 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" podStartSLOduration=2.557891293 podStartE2EDuration="2.557891293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:00.537038656 +0000 UTC m=+1393.506533646" watchObservedRunningTime="2026-10-30 05:07:00.557891293 +0000 UTC m=+1393.527386284" Oct 30 05:07:01.774026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.773991 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:07:01.779702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.779679 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:01.797134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.797108 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:07:01.803782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.803757 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-dfb231\"" Oct 30 05:07:01.841567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.841533 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:07:01.846368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.846346 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:01.864898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.864867 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-dfb231-dockercfg-dbm2x\"" Oct 30 05:07:01.867400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.867370 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:07:01.957303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.955156 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:01.957303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.956474 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:01.957303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.956524 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxd4\" (UniqueName: \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:01.957303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:01.956644 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg85l\" (UniqueName: \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.171242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.170084 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg85l\" (UniqueName: \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.171242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.170352 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:02.171242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.170456 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.171242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.170497 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxd4\" (UniqueName: \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:02.190246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.188528 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:02.190246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.189479 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.284065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.284030 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg85l\" (UniqueName: \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\") pod \"payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.305495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.284033 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxd4\" (UniqueName: \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\") pod \"payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:02.400545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.400491 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:02.467702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.465869 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:02.750696 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:07:02.750661 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc5c8fa_163f_4833_b402_8ea2a285b71a.slice/crio-f733c914bf3a4334a073fb6e0af91fb4db875173209a9d7945f52f6ab8870aa1 WatchSource:0}: Error finding container f733c914bf3a4334a073fb6e0af91fb4db875173209a9d7945f52f6ab8870aa1: Status 404 returned error can't find the container with id f733c914bf3a4334a073fb6e0af91fb4db875173209a9d7945f52f6ab8870aa1 Oct 30 05:07:02.752669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.752642 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:07:02.951788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:02.928709 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:07:03.493311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.493273 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" event={"ID":"6ba47046-1aca-4cff-afe0-56396502592a","Type":"ContainerStarted","Data":"32d7486602eb13f84fe0055143cb9c1e6eb7312d2586d0285401436560e91ac3"} Oct 30 05:07:03.493518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.493336 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" event={"ID":"6ba47046-1aca-4cff-afe0-56396502592a","Type":"ContainerStarted","Data":"5f5836e2eac0304d03a7d44404eefbf1e4ebfe4ced86932678e8ddd765d0a046"} Oct 30 05:07:03.493518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.493366 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:03.494993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.494969 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" event={"ID":"6fc5c8fa-163f-4833-b402-8ea2a285b71a","Type":"ContainerStarted","Data":"45aff2d006b7503ef4765a8b4af5fdacd42070e0db9d0a2424b161cae7d96c05"} Oct 30 05:07:03.495125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.495013 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" event={"ID":"6fc5c8fa-163f-4833-b402-8ea2a285b71a","Type":"ContainerStarted","Data":"f733c914bf3a4334a073fb6e0af91fb4db875173209a9d7945f52f6ab8870aa1"} Oct 30 05:07:03.495125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.495046 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:03.537929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.537721 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" podStartSLOduration=2.537705661 podStartE2EDuration="2.537705661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:03.534344219 +0000 UTC m=+1396.503839217" watchObservedRunningTime="2026-10-30 05:07:03.537705661 +0000 UTC m=+1396.507200651" Oct 30 05:07:03.576510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:03.576461 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" podStartSLOduration=2.576448224 podStartE2EDuration="2.576448224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:03.57479909 +0000 UTC m=+1396.544294080" watchObservedRunningTime="2026-10-30 05:07:03.576448224 +0000 UTC m=+1396.545943213" Oct 30 05:07:12.487844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:12.487811 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:07:14.531263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:14.530697 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:07:14.546243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:14.541013 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:07:19.193986 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:07:19.193935 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:07:46.792374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:46.792337 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-182852"] Oct 30 05:07:46.797730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:46.797705 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-182852" Oct 30 05:07:46.873544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:46.872568 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\"" Oct 30 05:07:46.901584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:46.901436 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-182852"] Oct 30 05:07:46.952091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:46.951561 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz554\" (UniqueName: \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\") pod \"test-curl-8867-182852\" (UID: \"858dd400-da19-4211-b50a-69a2f47f7990\") " pod="opendatahub/test-curl-8867-182852" Oct 30 05:07:47.053262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:47.053168 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz554\" (UniqueName: \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\") pod \"test-curl-8867-182852\" (UID: \"858dd400-da19-4211-b50a-69a2f47f7990\") " pod="opendatahub/test-curl-8867-182852" Oct 30 05:07:47.073626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:47.073592 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz554\" (UniqueName: \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\") pod \"test-curl-8867-182852\" (UID: \"858dd400-da19-4211-b50a-69a2f47f7990\") " pod="opendatahub/test-curl-8867-182852" Oct 30 05:07:47.113660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:47.113626 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-182852" Oct 30 05:07:47.424127 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:07:47.424088 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod858dd400_da19_4211_b50a_69a2f47f7990.slice/crio-8dec9eb5937f38463d1d434fea646f6915f68a8badc41ee8099ba09d48614797 WatchSource:0}: Error finding container 8dec9eb5937f38463d1d434fea646f6915f68a8badc41ee8099ba09d48614797: Status 404 returned error can't find the container with id 8dec9eb5937f38463d1d434fea646f6915f68a8badc41ee8099ba09d48614797 Oct 30 05:07:47.425720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:47.425694 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-182852"] Oct 30 05:07:47.510014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:47.509976 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-182852" event={"ID":"858dd400-da19-4211-b50a-69a2f47f7990","Type":"ContainerStarted","Data":"8dec9eb5937f38463d1d434fea646f6915f68a8badc41ee8099ba09d48614797"} Oct 30 05:07:49.192860 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:07:49.192813 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:07:55.030341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:55.030293 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:07:55.076239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:07:55.072851 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:01.438932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.438898 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:08:01.446544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.446519 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.469395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.468688 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:08:01.472630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.472605 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-543d08-dockercfg-7v82h\"" Oct 30 05:08:01.472755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.472636 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-543d08\"" Oct 30 05:08:01.610856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.610815 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhc5c\" (UniqueName: \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.636242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.631527 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.944706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.776949 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhc5c\" (UniqueName: \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.944706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.777103 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.944706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.778196 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:01.944706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.810360 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:08:01.944706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:01.894308 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.005307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.003149 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffklw\" (UniqueName: \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.005307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.003517 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.023639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.008117 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhc5c\" (UniqueName: \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\") pod \"payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:02.023639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.016683 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:02.048552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.037618 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:08:02.105281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.105243 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffklw\" (UniqueName: \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.105505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.105484 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.106663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.106575 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.133286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.129797 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffklw\" (UniqueName: \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\") pod \"payload-processing-e2e-api-a-543d08-6787968b78-jpdvp\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.407229 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:02.407179 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d58a1f9_0cf4_480f_8238_c60f9f3fd092.slice/crio-8a14bc6bfe34d9a8a22ff3e804de3b8497a74a02be210f5ce97ea4420b83f4a4 WatchSource:0}: Error finding container 8a14bc6bfe34d9a8a22ff3e804de3b8497a74a02be210f5ce97ea4420b83f4a4: Status 404 returned error can't find the container with id 8a14bc6bfe34d9a8a22ff3e804de3b8497a74a02be210f5ce97ea4420b83f4a4 Oct 30 05:08:02.413131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.413097 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:08:02.433915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.426356 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:02.783905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:02.783873 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:08:02.784528 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:02.784494 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09054a92_a469_42a5_a610_ce9b9042d503.slice/crio-0e719ae3e83f936214404179870b9412061c6ea412bbf342e436a4f772f740b2 WatchSource:0}: Error finding container 0e719ae3e83f936214404179870b9412061c6ea412bbf342e436a4f772f740b2: Status 404 returned error can't find the container with id 0e719ae3e83f936214404179870b9412061c6ea412bbf342e436a4f772f740b2 Oct 30 05:08:03.135305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.135145 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" event={"ID":"6d58a1f9-0cf4-480f-8238-c60f9f3fd092","Type":"ContainerStarted","Data":"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff"} Oct 30 05:08:03.135305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.135239 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" event={"ID":"6d58a1f9-0cf4-480f-8238-c60f9f3fd092","Type":"ContainerStarted","Data":"8a14bc6bfe34d9a8a22ff3e804de3b8497a74a02be210f5ce97ea4420b83f4a4"} Oct 30 05:08:03.143701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.136768 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:03.153243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.151291 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" event={"ID":"09054a92-a469-42a5-a610-ce9b9042d503","Type":"ContainerStarted","Data":"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417"} Oct 30 05:08:03.153243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.151358 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" event={"ID":"09054a92-a469-42a5-a610-ce9b9042d503","Type":"ContainerStarted","Data":"0e719ae3e83f936214404179870b9412061c6ea412bbf342e436a4f772f740b2"} Oct 30 05:08:03.153243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.152622 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:03.220244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:03.218582 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" podStartSLOduration=2.218559832 podStartE2EDuration="2.218559832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:03.198169933 +0000 UTC m=+1456.167664922" watchObservedRunningTime="2026-10-30 05:08:03.218559832 +0000 UTC m=+1456.188054826" Oct 30 05:08:05.304863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:05.304828 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-182852" event={"ID":"858dd400-da19-4211-b50a-69a2f47f7990","Type":"ContainerStarted","Data":"4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0"} Oct 30 05:08:05.347945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:05.347826 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" podStartSLOduration=4.347807529 podStartE2EDuration="4.347807529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:03.295725612 +0000 UTC m=+1456.265220602" watchObservedRunningTime="2026-10-30 05:08:05.347807529 +0000 UTC m=+1458.317302523" Oct 30 05:08:05.349260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:05.349205 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-182852" podStartSLOduration=1.695231149 podStartE2EDuration="19.349195918s" totalImagesPullingTime="17.653964769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:05.345211112 +0000 UTC m=+1458.314706100" watchObservedRunningTime="2026-10-30 05:08:05.349195918 +0000 UTC m=+1458.318690908" Oct 30 05:08:06.315125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.315092 5058 generic.go:356] "Generic (PLEG): container finished" podID="25d26a26-b1d8-4390-8883-7965d19497d4" containerID="4adb03779991e646532b9b1da0bfef5007433f444adc4ed5a573949b7281da85" exitCode=0 Oct 30 05:08:06.315612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.315164 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerDied","Data":"4adb03779991e646532b9b1da0bfef5007433f444adc4ed5a573949b7281da85"} Oct 30 05:08:06.315700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.315672 5058 scope.go:122] "RemoveContainer" containerID="4adb03779991e646532b9b1da0bfef5007433f444adc4ed5a573949b7281da85" Oct 30 05:08:06.317432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.317405 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:06.317528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.317484 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:06.317565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.317533 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:06.335151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.335118 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-182852"] Oct 30 05:08:06.349092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.349060 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-182852"] Oct 30 05:08:06.492984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.492419 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-4aad04"] Oct 30 05:08:06.514854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.493829 5058 cpu_manager.go:414] "containerMap: removing container" podUID="858dd400-da19-4211-b50a-69a2f47f7990" containerName="test-curl-8867-182852" Oct 30 05:08:06.514854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.493855 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858dd400-da19-4211-b50a-69a2f47f7990" containerName="test-curl-8867-182852" Oct 30 05:08:06.514854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.494021 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="858dd400-da19-4211-b50a-69a2f47f7990" containerName="test-curl-8867-182852" Oct 30 05:08:06.514854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.511512 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:06.521382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.521348 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-4aad04"] Oct 30 05:08:06.598637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.598544 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswwp\" (UniqueName: \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\") pod \"test-curl-8867-4aad04\" (UID: \"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926\") " pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:06.700514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.700482 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswwp\" (UniqueName: \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\") pod \"test-curl-8867-4aad04\" (UID: \"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926\") " pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:06.721331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.721299 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswwp\" (UniqueName: \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\") pod \"test-curl-8867-4aad04\" (UID: \"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926\") " pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:06.831813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:06.831779 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:07.168609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.168587 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-4aad04"] Oct 30 05:08:07.404864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.404758 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerStarted","Data":"5c148558e63780fdb591852e220305ae9c6e44d50bec3e1febe3cbc077e886be"} Oct 30 05:08:07.417020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.416957 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-182852" podUID="858dd400-da19-4211-b50a-69a2f47f7990" containerName="test-curl-8867-182852" containerID="cri-o://4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0" gracePeriod=2 Oct 30 05:08:07.417526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.417488 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-4aad04" event={"ID":"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926","Type":"ContainerStarted","Data":"5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4"} Oct 30 05:08:07.417704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.417689 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-4aad04" event={"ID":"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926","Type":"ContainerStarted","Data":"9e4f5ab5c8e10636532b1f4020f074c9853ae8fe7b03a0086e16ec97e269a7f3"} Oct 30 05:08:07.557268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.552880 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-4aad04" podStartSLOduration=1.5528592940000001 podStartE2EDuration="1.552859294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:07.552817735 +0000 UTC m=+1460.522312724" watchObservedRunningTime="2026-10-30 05:08:07.552859294 +0000 UTC m=+1460.522354284" Oct 30 05:08:07.928243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.927044 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:08:07.928243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.927277 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:07.928243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:07.927348 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:08.593685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.593648 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-4aad04"] Oct 30 05:08:08.605799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.605762 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-4aad04"] Oct 30 05:08:08.757842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.757813 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-280ef2"] Oct 30 05:08:08.758786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.758758 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" containerName="test-curl-8867-4aad04" Oct 30 05:08:08.758786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.758786 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" containerName="test-curl-8867-4aad04" Oct 30 05:08:08.759023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.759008 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" containerName="test-curl-8867-4aad04" Oct 30 05:08:08.766334 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.766311 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:08.777617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.777274 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-280ef2"] Oct 30 05:08:08.918128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.918087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpt4f\" (UniqueName: \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\") pod \"test-curl-8867-280ef2\" (UID: \"2ff13395-92e1-49d4-8b3f-948edaf8c397\") " pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:08.921728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.921694 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:08.921866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:08.921759 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:09.020306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.020269 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt4f\" (UniqueName: \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\") pod \"test-curl-8867-280ef2\" (UID: \"2ff13395-92e1-49d4-8b3f-948edaf8c397\") " pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:09.051894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.051848 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt4f\" (UniqueName: \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\") pod \"test-curl-8867-280ef2\" (UID: \"2ff13395-92e1-49d4-8b3f-948edaf8c397\") " pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:09.083433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.083402 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:09.416363 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:09.416326 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ff13395_92e1_49d4_8b3f_948edaf8c397.slice/crio-e88890683360a5f69d4f30add02b6e48a5dd61320f674fdbd34868f09619ac90 WatchSource:0}: Error finding container e88890683360a5f69d4f30add02b6e48a5dd61320f674fdbd34868f09619ac90: Status 404 returned error can't find the container with id e88890683360a5f69d4f30add02b6e48a5dd61320f674fdbd34868f09619ac90 Oct 30 05:08:09.418162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.418141 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-280ef2"] Oct 30 05:08:09.461308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.461272 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-280ef2" event={"ID":"2ff13395-92e1-49d4-8b3f-948edaf8c397","Type":"ContainerStarted","Data":"e88890683360a5f69d4f30add02b6e48a5dd61320f674fdbd34868f09619ac90"} Oct 30 05:08:09.461502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.461475 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-4aad04" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" containerName="test-curl-8867-4aad04" containerID="cri-o://5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4" gracePeriod=2 Oct 30 05:08:09.923253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.921462 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:09.923253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.921520 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:09.975244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:09.972501 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-182852" Oct 30 05:08:10.009237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.008254 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\" (UniqueName: \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\") pod \"858dd400-da19-4211-b50a-69a2f47f7990\" (UID: \"858dd400-da19-4211-b50a-69a2f47f7990\") " Oct 30 05:08:10.017928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.017888 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554" pod "858dd400-da19-4211-b50a-69a2f47f7990" (UID: "858dd400-da19-4211-b50a-69a2f47f7990"). InnerVolumeSpecName "kube-api-access-xz554". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:10.114143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.114110 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz554\" (UniqueName: \"kubernetes.io/projected/858dd400-da19-4211-b50a-69a2f47f7990-kube-api-access-xz554\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:10.486437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.486398 5058 generic.go:356] "Generic (PLEG): container finished" podID="858dd400-da19-4211-b50a-69a2f47f7990" containerID="4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0" exitCode=137 Oct 30 05:08:10.486979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.486675 5058 scope.go:122] "RemoveContainer" containerID="4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0" Oct 30 05:08:10.487691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.486736 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-182852" Oct 30 05:08:10.495601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.495569 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-280ef2" event={"ID":"2ff13395-92e1-49d4-8b3f-948edaf8c397","Type":"ContainerStarted","Data":"242b2751abec56c385c3f9b5e42ca4ea9cfa5cab2f4921a82d97e7e2c2866a38"} Oct 30 05:08:10.504521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.504494 5058 scope.go:122] "RemoveContainer" containerID="4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0" Oct 30 05:08:10.504940 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:10.504896 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0\": container with ID starting with 4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0 not found: ID does not exist" containerID="4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0" Oct 30 05:08:10.505035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.504957 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0"} err="failed to get container status \"4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0\": rpc error: code = NotFound desc = could not find container \"4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0\": container with ID starting with 4e7ed43ab072ad17974e9aef903571be5f739efa4abd315cc711ca5bcf60e5d0 not found: ID does not exist" Oct 30 05:08:10.554494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.546086 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-280ef2" podStartSLOduration=2.5460619429999998 podStartE2EDuration="2.546061943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:10.536577842 +0000 UTC m=+1463.506072833" watchObservedRunningTime="2026-10-30 05:08:10.546061943 +0000 UTC m=+1463.515556933" Oct 30 05:08:10.554705 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:10.553612 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-182852\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="858dd400-da19-4211-b50a-69a2f47f7990" pod="opendatahub/test-curl-8867-182852" Oct 30 05:08:10.921525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.921488 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:10.921701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:10.921549 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:11.169696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.169657 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858dd400-da19-4211-b50a-69a2f47f7990" path="/var/lib/kubelet/pods/858dd400-da19-4211-b50a-69a2f47f7990/volumes" Oct 30 05:08:11.506417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.506147 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-280ef2"] Oct 30 05:08:11.611598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.611563 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-280ef2"] Oct 30 05:08:11.618101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.618051 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-fbfc64"] Oct 30 05:08:11.619603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.619563 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" containerName="test-curl-8867-280ef2" Oct 30 05:08:11.619603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.619591 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" containerName="test-curl-8867-280ef2" Oct 30 05:08:11.619860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.619846 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" containerName="test-curl-8867-280ef2" Oct 30 05:08:11.657634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.657589 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-fbfc64"] Oct 30 05:08:11.657840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.657747 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:11.798249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.798198 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgq6\" (UniqueName: \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\") pod \"test-curl-8867-fbfc64\" (UID: \"6d70ea6e-e55d-45e2-b662-ccee13a6b566\") " pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:11.900298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.900266 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgq6\" (UniqueName: \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\") pod \"test-curl-8867-fbfc64\" (UID: \"6d70ea6e-e55d-45e2-b662-ccee13a6b566\") " pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:11.910749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.910720 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:11.920645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.920619 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgq6\" (UniqueName: \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\") pod \"test-curl-8867-fbfc64\" (UID: \"6d70ea6e-e55d-45e2-b662-ccee13a6b566\") " pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:11.921615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.921586 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:11.921720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.921637 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:11.984905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:11.984869 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:12.002473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.002213 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\" (UniqueName: \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\") pod \"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926\" (UID: \"ac2a3ff9-98ae-4578-965c-7c8e5b3ed926\") " Oct 30 05:08:12.006511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.006457 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp" pod "ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" (UID: "ac2a3ff9-98ae-4578-965c-7c8e5b3ed926"). InnerVolumeSpecName "kube-api-access-qswwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:12.104894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.104845 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswwp\" (UniqueName: \"kubernetes.io/projected/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926-kube-api-access-qswwp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:12.321956 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:12.321923 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d70ea6e_e55d_45e2_b662_ccee13a6b566.slice/crio-a3d8ea39a84b0fa81ebf8a810ed7a7a2d95e688d8bc47e9c179b4ecb821ea863 WatchSource:0}: Error finding container a3d8ea39a84b0fa81ebf8a810ed7a7a2d95e688d8bc47e9c179b4ecb821ea863: Status 404 returned error can't find the container with id a3d8ea39a84b0fa81ebf8a810ed7a7a2d95e688d8bc47e9c179b4ecb821ea863 Oct 30 05:08:12.352533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.352498 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-fbfc64"] Oct 30 05:08:12.606301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.602594 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-fbfc64" event={"ID":"6d70ea6e-e55d-45e2-b662-ccee13a6b566","Type":"ContainerStarted","Data":"19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001"} Oct 30 05:08:12.606301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.602764 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-fbfc64" event={"ID":"6d70ea6e-e55d-45e2-b662-ccee13a6b566","Type":"ContainerStarted","Data":"a3d8ea39a84b0fa81ebf8a810ed7a7a2d95e688d8bc47e9c179b4ecb821ea863"} Oct 30 05:08:12.624250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.622894 5058 generic.go:356] "Generic (PLEG): container finished" podID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" containerID="5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4" exitCode=137 Oct 30 05:08:12.624250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.623248 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-280ef2" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" containerName="test-curl-8867-280ef2" containerID="cri-o://242b2751abec56c385c3f9b5e42ca4ea9cfa5cab2f4921a82d97e7e2c2866a38" gracePeriod=2 Oct 30 05:08:12.624250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.623618 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:12.628359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.625517 5058 scope.go:122] "RemoveContainer" containerID="5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4" Oct 30 05:08:12.646667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.645720 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-fbfc64" podStartSLOduration=1.645698874 podStartE2EDuration="1.645698874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:12.64381892 +0000 UTC m=+1465.613313909" watchObservedRunningTime="2026-10-30 05:08:12.645698874 +0000 UTC m=+1465.615193863" Oct 30 05:08:12.671360 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:12.669572 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-280ef2\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:12.744826 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:12.693472 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-4aad04\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" pod="opendatahub/test-curl-8867-4aad04" Oct 30 05:08:12.790370 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:12.768536 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-280ef2\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:12.790370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.788920 5058 scope.go:122] "RemoveContainer" containerID="5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4" Oct 30 05:08:12.790910 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:12.790673 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4\": container with ID starting with 5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4 not found: ID does not exist" containerID="5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4" Oct 30 05:08:12.790910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.790715 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4"} err="failed to get container status \"5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4\": rpc error: code = NotFound desc = could not find container \"5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4\": container with ID starting with 5a2ad631ec5f2de747e5181f3cdd84944a193b4c40e1fdd532ebf321711967a4 not found: ID does not exist" Oct 30 05:08:12.921389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.921352 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 05:08:12.921570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:12.921420 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 05:08:13.168374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:13.168333 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2a3ff9-98ae-4578-965c-7c8e5b3ed926" path="/var/lib/kubelet/pods/ac2a3ff9-98ae-4578-965c-7c8e5b3ed926/volumes" Oct 30 05:08:13.333353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:13.329759 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:08:13.938911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:13.934137 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:08:13.958694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:13.955702 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:08:14.116165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:14.116128 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-fbfc64"] Oct 30 05:08:14.122821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:14.122771 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-fbfc64"] Oct 30 05:08:14.777656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:14.777616 5058 generic.go:356] "Generic (PLEG): container finished" podID="2ff13395-92e1-49d4-8b3f-948edaf8c397" containerID="242b2751abec56c385c3f9b5e42ca4ea9cfa5cab2f4921a82d97e7e2c2866a38" exitCode=137 Oct 30 05:08:14.778202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:14.778114 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-fbfc64" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" containerName="test-curl-8867-fbfc64" containerID="cri-o://19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001" gracePeriod=2 Oct 30 05:08:14.798564 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:14.798506 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-fbfc64\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:15.003337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.003304 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:15.018198 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:15.018157 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-fbfc64\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:15.036739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.036706 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\" (UniqueName: \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\") pod \"2ff13395-92e1-49d4-8b3f-948edaf8c397\" (UID: \"2ff13395-92e1-49d4-8b3f-948edaf8c397\") " Oct 30 05:08:15.038787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.038763 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f" pod "2ff13395-92e1-49d4-8b3f-948edaf8c397" (UID: "2ff13395-92e1-49d4-8b3f-948edaf8c397"). InnerVolumeSpecName "kube-api-access-lpt4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:15.138919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.138880 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpt4f\" (UniqueName: \"kubernetes.io/projected/2ff13395-92e1-49d4-8b3f-948edaf8c397-kube-api-access-lpt4f\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:15.164040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.163968 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff13395-92e1-49d4-8b3f-948edaf8c397" path="/var/lib/kubelet/pods/2ff13395-92e1-49d4-8b3f-948edaf8c397/volumes" Oct 30 05:08:15.308889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.308862 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:08:15.309061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.308928 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:08:15.788663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.788634 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-280ef2" Oct 30 05:08:15.789118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:15.788640 5058 scope.go:122] "RemoveContainer" containerID="242b2751abec56c385c3f9b5e42ca4ea9cfa5cab2f4921a82d97e7e2c2866a38" Oct 30 05:08:16.941242 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:16.939788 5058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d70ea6e_e55d_45e2_b662_ccee13a6b566.slice/crio-conmon-19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:08:17.255136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.255108 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:17.270012 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:17.269974 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-fbfc64\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:17.339012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.338979 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\" (UniqueName: \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\") pod \"6d70ea6e-e55d-45e2-b662-ccee13a6b566\" (UID: \"6d70ea6e-e55d-45e2-b662-ccee13a6b566\") " Oct 30 05:08:17.341274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.341207 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6" pod "6d70ea6e-e55d-45e2-b662-ccee13a6b566" (UID: "6d70ea6e-e55d-45e2-b662-ccee13a6b566"). InnerVolumeSpecName "kube-api-access-6fgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:17.441349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.441317 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgq6\" (UniqueName: \"kubernetes.io/projected/6d70ea6e-e55d-45e2-b662-ccee13a6b566-kube-api-access-6fgq6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:17.819770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.819685 5058 generic.go:356] "Generic (PLEG): container finished" podID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" containerID="19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001" exitCode=137 Oct 30 05:08:17.819770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.819762 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:17.819972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.819785 5058 scope.go:122] "RemoveContainer" containerID="19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001" Oct 30 05:08:17.830947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.830921 5058 scope.go:122] "RemoveContainer" containerID="19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001" Oct 30 05:08:17.831270 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:17.831250 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001\": container with ID starting with 19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001 not found: ID does not exist" containerID="19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001" Oct 30 05:08:17.831338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:17.831283 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001"} err="failed to get container status \"19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001\": rpc error: code = NotFound desc = could not find container \"19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001\": container with ID starting with 19affea8d56e63dc2d3f42996a68568b5897111211febf6a461e0cb70ee2b001 not found: ID does not exist" Oct 30 05:08:17.834700 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:17.834672 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-fbfc64\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:17.849404 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:17.849373 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-fbfc64\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" pod="opendatahub/test-curl-8867-fbfc64" Oct 30 05:08:19.163686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:19.163654 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" path="/var/lib/kubelet/pods/6d70ea6e-e55d-45e2-b662-ccee13a6b566/volumes" Oct 30 05:08:19.195698 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:19.195658 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:08:23.943242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:23.942382 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 05:08:23.960242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:23.955501 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" containerID="cri-o://2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771" gracePeriod=30 Oct 30 05:08:23.992657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:23.992426 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:08:23.992836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:23.992690 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" containerID="cri-o://9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8" gracePeriod=30 Oct 30 05:08:24.002528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:24.002098 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:08:24.002528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:24.002336 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" containerID="cri-o://6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013" gracePeriod=30 Oct 30 05:08:24.021098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:24.017393 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:08:24.021098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:24.017639 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" containerID="cri-o://0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23" gracePeriod=30 Oct 30 05:08:29.417270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.417204 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 05:08:29.472970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.472928 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:08:29.485457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.485423 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:08:29.490764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.490738 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:08:29.513963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.513931 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.513972 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514000 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514038 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514077 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514102 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.514373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514134 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.514373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514204 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.514373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514258 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.514373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514324 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514375 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514407 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514432 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514482 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514541 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.514580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514573 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.514932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514639 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.514932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514700 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.514932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514769 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.515366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514802 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.515455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515411 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.515455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515447 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.515577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515499 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.515577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515537 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.515673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515598 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.515673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515628 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.515673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515656 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.515826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514320 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.515826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515804 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.515826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514438 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.514523 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515042 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515842 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515903 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\") pod \"278896e4-e88d-419e-97b2-31bcef5426ee\" (UID: \"278896e4-e88d-419e-97b2-31bcef5426ee\") " Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515933 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\") pod \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\" (UID: \"451878c2-d0e7-4e41-8cab-80d0bd8f57d1\") " Oct 30 05:08:29.515974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515962 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.516299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516008 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.516299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516121 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.516299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516150 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\") pod \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\" (UID: \"b49afaaf-b98e-49c2-af0e-d5316057cb6b\") " Oct 30 05:08:29.516299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516180 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\") pod \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\" (UID: \"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd\") " Oct 30 05:08:29.517374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517349 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.517488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517379 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.517488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517394 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.517488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517410 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.519378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.515761 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.519502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516659 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.519590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516791 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.519674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.516908 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.519790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517647 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.519872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.517827 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.519959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518085 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518459 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.520098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518520 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.520181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518660 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518795 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518830 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.520497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518882 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518897 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.518999 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.520742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.519099 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.522820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.519348 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.522930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.522246 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.524530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.524482 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.526723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.526677 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.526955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.526931 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.527061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.526960 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "kube-api-access-7xzcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.527061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.526983 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.527295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.527275 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.528383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.528357 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.528660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.528379 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy" pod "278896e4-e88d-419e-97b2-31bcef5426ee" (UID: "278896e4-e88d-419e-97b2-31bcef5426ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.528660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.528460 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.528660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.528613 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "kube-api-access-8v9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.529887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.529811 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy" pod "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" (UID: "aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.530498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.530441 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "kube-api-access-f97qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.530498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.530478 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh" pod "b49afaaf-b98e-49c2-af0e-d5316057cb6b" (UID: "b49afaaf-b98e-49c2-af0e-d5316057cb6b"). InnerVolumeSpecName "kube-api-access-np8hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.530498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.530476 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token" pod "451878c2-d0e7-4e41-8cab-80d0bd8f57d1" (UID: "451878c2-d0e7-4e41-8cab-80d0bd8f57d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619050 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619080 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzcp\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-kube-api-access-7xzcp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619092 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619101 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619112 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619119 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619127 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619135 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619146 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619154 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619162 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619171 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8hh\" (UniqueName: \"kubernetes.io/projected/b49afaaf-b98e-49c2-af0e-d5316057cb6b-kube-api-access-np8hh\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619179 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619190 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619203 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619212 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619235 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619245 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619253 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619263 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278896e4-e88d-419e-97b2-31bcef5426ee-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619273 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619284 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9c7\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-kube-api-access-8v9c7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619293 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619303 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278896e4-e88d-419e-97b2-31bcef5426ee-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619311 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619319 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49afaaf-b98e-49c2-af0e-d5316057cb6b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619328 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619335 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278896e4-e88d-419e-97b2-31bcef5426ee-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619342 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619351 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278896e4-e88d-419e-97b2-31bcef5426ee-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619362 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97qr\" (UniqueName: \"kubernetes.io/projected/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-kube-api-access-f97qr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.619666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:29.619370 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451878c2-d0e7-4e41-8cab-80d0bd8f57d1-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:30.022111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.022048 5058 generic.go:356] "Generic (PLEG): container finished" podID="278896e4-e88d-419e-97b2-31bcef5426ee" containerID="6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013" exitCode=0 Oct 30 05:08:30.022111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.022098 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" Oct 30 05:08:30.022391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.022148 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" event={"ID":"278896e4-e88d-419e-97b2-31bcef5426ee","Type":"ContainerDied","Data":"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013"} Oct 30 05:08:30.022391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.022206 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898" event={"ID":"278896e4-e88d-419e-97b2-31bcef5426ee","Type":"ContainerDied","Data":"b86fe3f7aa5241e2b5f99a8a84195d82ca40fabbd01a3776912625251cc1e539"} Oct 30 05:08:30.022391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.022237 5058 scope.go:122] "RemoveContainer" containerID="6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013" Oct 30 05:08:30.024174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.024141 5058 generic.go:356] "Generic (PLEG): container finished" podID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerID="0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23" exitCode=0 Oct 30 05:08:30.024440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.024195 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" Oct 30 05:08:30.024440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.024205 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" event={"ID":"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd","Type":"ContainerDied","Data":"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23"} Oct 30 05:08:30.024440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.024368 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj" event={"ID":"aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd","Type":"ContainerDied","Data":"51ebf407e46c0ef1bf3f41d086f94a748c200aaabe316ab13f27019b5f8942c2"} Oct 30 05:08:30.027851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.027737 5058 generic.go:356] "Generic (PLEG): container finished" podID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerID="2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771" exitCode=0 Oct 30 05:08:30.027851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.027844 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerDied","Data":"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771"} Oct 30 05:08:30.028285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.027876 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" event={"ID":"451878c2-d0e7-4e41-8cab-80d0bd8f57d1","Type":"ContainerDied","Data":"f2cf04991024b38609587a3edfb5f3616357edf50195f8e96acc1319ee9d5166"} Oct 30 05:08:30.028285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.027932 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb" Oct 30 05:08:30.031761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.031736 5058 generic.go:356] "Generic (PLEG): container finished" podID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerID="9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8" exitCode=0 Oct 30 05:08:30.031880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.031827 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" event={"ID":"b49afaaf-b98e-49c2-af0e-d5316057cb6b","Type":"ContainerDied","Data":"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8"} Oct 30 05:08:30.031880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.031869 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" event={"ID":"b49afaaf-b98e-49c2-af0e-d5316057cb6b","Type":"ContainerDied","Data":"a1fb1e37d84a4c4cd4de7f3881e2c829cdbb6ae7d19b2f0f60ab25623cd165c2"} Oct 30 05:08:30.031986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.031943 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t" Oct 30 05:08:30.058903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.058879 5058 scope.go:122] "RemoveContainer" containerID="6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013" Oct 30 05:08:30.059324 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:30.059291 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013\": container with ID starting with 6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013 not found: ID does not exist" containerID="6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013" Oct 30 05:08:30.059398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.059337 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013"} err="failed to get container status \"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013\": rpc error: code = NotFound desc = could not find container \"6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013\": container with ID starting with 6e732d7ee7a4d7d74c0adf0e9582572a3896cbcd578007802354e2f403569013 not found: ID does not exist" Oct 30 05:08:30.059398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.059356 5058 scope.go:122] "RemoveContainer" containerID="0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23" Oct 30 05:08:30.070577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.070554 5058 scope.go:122] "RemoveContainer" containerID="0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23" Oct 30 05:08:30.070974 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:30.070939 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23\": container with ID starting with 0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23 not found: ID does not exist" containerID="0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23" Oct 30 05:08:30.071071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.070982 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23"} err="failed to get container status \"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23\": rpc error: code = NotFound desc = could not find container \"0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23\": container with ID starting with 0360f3b212e30de48f008aef9d9ef8dfb29e86e92daa7dfdcb26a44e2c6d7c23 not found: ID does not exist" Oct 30 05:08:30.071071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.071007 5058 scope.go:122] "RemoveContainer" containerID="2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771" Oct 30 05:08:30.081793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.081772 5058 scope.go:122] "RemoveContainer" containerID="90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704" Oct 30 05:08:30.092313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.092291 5058 scope.go:122] "RemoveContainer" containerID="2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771" Oct 30 05:08:30.092643 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:30.092624 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771\": container with ID starting with 2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771 not found: ID does not exist" containerID="2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771" Oct 30 05:08:30.092699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.092652 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771"} err="failed to get container status \"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771\": rpc error: code = NotFound desc = could not find container \"2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771\": container with ID starting with 2f53c85e4270e1656fa2a49095cf3ec29bdb6fda3d90ea50df3165f407d0b771 not found: ID does not exist" Oct 30 05:08:30.092699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.092674 5058 scope.go:122] "RemoveContainer" containerID="90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704" Oct 30 05:08:30.092942 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:30.092913 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704\": container with ID starting with 90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704 not found: ID does not exist" containerID="90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704" Oct 30 05:08:30.093042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.092948 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704"} err="failed to get container status \"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704\": rpc error: code = NotFound desc = could not find container \"90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704\": container with ID starting with 90d3352f4cbc1e9c8a7c6b2d8ed9dce955915b0053d7447b766a510c450f1704 not found: ID does not exist" Oct 30 05:08:30.093042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.092965 5058 scope.go:122] "RemoveContainer" containerID="9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8" Oct 30 05:08:30.106636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.106617 5058 scope.go:122] "RemoveContainer" containerID="9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8" Oct 30 05:08:30.107197 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:30.107170 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8\": container with ID starting with 9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8 not found: ID does not exist" containerID="9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8" Oct 30 05:08:30.107363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.107203 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8"} err="failed to get container status \"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8\": rpc error: code = NotFound desc = could not find container \"9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8\": container with ID starting with 9f3ec3d53fc03a827e6bafa8b175aa8660e9d1252a6a44281f31d3a1990597f8 not found: ID does not exist" Oct 30 05:08:30.134622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.134588 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:08:30.147459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.147428 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj"] Oct 30 05:08:30.177589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.177557 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:08:30.201756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.201539 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898"] Oct 30 05:08:30.362138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.308025 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:08:30.362138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.357117 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t"] Oct 30 05:08:30.378411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.378320 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 05:08:30.382787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:30.382746 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb"] Oct 30 05:08:31.164105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.164067 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" path="/var/lib/kubelet/pods/278896e4-e88d-419e-97b2-31bcef5426ee/volumes" Oct 30 05:08:31.164687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.164670 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" path="/var/lib/kubelet/pods/451878c2-d0e7-4e41-8cab-80d0bd8f57d1/volumes" Oct 30 05:08:31.165152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.165138 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" path="/var/lib/kubelet/pods/aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd/volumes" Oct 30 05:08:31.165982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.165961 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" path="/var/lib/kubelet/pods/b49afaaf-b98e-49c2-af0e-d5316057cb6b/volumes" Oct 30 05:08:31.260366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.260330 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:08:31.260626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.260601 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" podUID="540aa781-d6af-4ad1-827b-c09551d77cab" containerName="maas-api" containerID="cri-o://46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e" gracePeriod=30 Oct 30 05:08:31.827861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.818427 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:08:31.942240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.937058 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\") pod \"540aa781-d6af-4ad1-827b-c09551d77cab\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " Oct 30 05:08:31.942240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.937264 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\" (UniqueName: \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\") pod \"540aa781-d6af-4ad1-827b-c09551d77cab\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " Oct 30 05:08:31.942240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.937387 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\") pod \"540aa781-d6af-4ad1-827b-c09551d77cab\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " Oct 30 05:08:31.942240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.937471 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\") pod \"540aa781-d6af-4ad1-827b-c09551d77cab\" (UID: \"540aa781-d6af-4ad1-827b-c09551d77cab\") " Oct 30 05:08:31.965303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.958637 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca" pod "540aa781-d6af-4ad1-827b-c09551d77cab" (UID: "540aa781-d6af-4ad1-827b-c09551d77cab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:31.965303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.959408 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs" pod "540aa781-d6af-4ad1-827b-c09551d77cab" (UID: "540aa781-d6af-4ad1-827b-c09551d77cab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:08:31.965303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.959728 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls" pod "540aa781-d6af-4ad1-827b-c09551d77cab" (UID: "540aa781-d6af-4ad1-827b-c09551d77cab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:08:31.965303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:31.959886 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn" pod "540aa781-d6af-4ad1-827b-c09551d77cab" (UID: "540aa781-d6af-4ad1-827b-c09551d77cab"). InnerVolumeSpecName "kube-api-access-t7fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:32.158243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.152868 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540aa781-d6af-4ad1-827b-c09551d77cab-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:32.158243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.152909 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:32.158243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.152930 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fqn\" (UniqueName: \"kubernetes.io/projected/540aa781-d6af-4ad1-827b-c09551d77cab-kube-api-access-t7fqn\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:32.158243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.152943 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540aa781-d6af-4ad1-827b-c09551d77cab-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:32.259187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.258987 5058 generic.go:356] "Generic (PLEG): container finished" podID="540aa781-d6af-4ad1-827b-c09551d77cab" containerID="46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e" exitCode=0 Oct 30 05:08:32.260058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.259815 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" event={"ID":"540aa781-d6af-4ad1-827b-c09551d77cab","Type":"ContainerDied","Data":"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e"} Oct 30 05:08:32.260058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.259869 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" Oct 30 05:08:32.260058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.259876 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw" event={"ID":"540aa781-d6af-4ad1-827b-c09551d77cab","Type":"ContainerDied","Data":"ce989a9e66355f92a7c45a61e67ce1ebcdaa6dc1c2ddebbe60db5ad918807786"} Oct 30 05:08:32.260058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.259904 5058 scope.go:122] "RemoveContainer" containerID="46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e" Oct 30 05:08:32.275951 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.275928 5058 scope.go:122] "RemoveContainer" containerID="46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e" Oct 30 05:08:32.276279 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:32.276254 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e\": container with ID starting with 46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e not found: ID does not exist" containerID="46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e" Oct 30 05:08:32.276346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.276289 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e"} err="failed to get container status \"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e\": rpc error: code = NotFound desc = could not find container \"46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e\": container with ID starting with 46a507828c7c8f19a0c95412454df0646aa5396b5d434c0e83e630805fa0ab4e not found: ID does not exist" Oct 30 05:08:32.323423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.323390 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:08:32.330290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:32.330256 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-7fgxw"] Oct 30 05:08:33.163483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:33.163452 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540aa781-d6af-4ad1-827b-c09551d77cab" path="/var/lib/kubelet/pods/540aa781-d6af-4ad1-827b-c09551d77cab/volumes" Oct 30 05:08:36.232922 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.232888 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:08:36.233909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233885 5058 cpu_manager.go:414] "containerMap: removing container" podUID="540aa781-d6af-4ad1-827b-c09551d77cab" containerName="maas-api" Oct 30 05:08:36.233909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233909 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540aa781-d6af-4ad1-827b-c09551d77cab" containerName="maas-api" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233928 5058 cpu_manager.go:414] "containerMap: removing container" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233935 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233961 5058 cpu_manager.go:414] "containerMap: removing container" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233969 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233979 5058 cpu_manager.go:414] "containerMap: removing container" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.233985 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234022 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234031 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234053 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" containerName="test-curl-8867-fbfc64" Oct 30 05:08:36.234065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234060 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" containerName="test-curl-8867-fbfc64" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234262 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="278896e4-e88d-419e-97b2-31bcef5426ee" containerName="istio-proxy" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234284 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234293 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5e80c4-0c2b-4a1a-9ee0-5159a68368dd" containerName="istio-proxy" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234309 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49afaaf-b98e-49c2-af0e-d5316057cb6b" containerName="istio-proxy" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234320 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d70ea6e-e55d-45e2-b662-ccee13a6b566" containerName="test-curl-8867-fbfc64" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234328 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:36.234535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.234345 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="540aa781-d6af-4ad1-827b-c09551d77cab" containerName="maas-api" Oct 30 05:08:36.239818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.239796 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.254865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.254838 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:08:36.263794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.263747 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-543d08-serving-cert\"" Oct 30 05:08:36.275759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.275719 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9lc\" (UniqueName: \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.276067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.276043 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.276210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.276188 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.276309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.276281 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.377919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.377883 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.378076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.377945 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9lc\" (UniqueName: \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.378076 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:36.378045 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-serving-cert: secret "maas-api-e2e-api-a-543d08-serving-cert" not found Oct 30 05:08:36.378162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.378100 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.378162 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:36.378128 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls podName:3445476a-8e09-4468-864c-8be7aaa526e9 nodeName:}" failed. No retries permitted until 2026-10-30 05:08:36.878108174 +0000 UTC m=+1489.847603146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls") pod "maas-api-e2e-api-a-543d08-6f787759db-nsvhh" (UID: "3445476a-8e09-4468-864c-8be7aaa526e9") : secret "maas-api-e2e-api-a-543d08-serving-cert" not found Oct 30 05:08:36.378260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.378247 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.378929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.378907 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.380605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.380585 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.399210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.399181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9lc\" (UniqueName: \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.923276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.918591 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:36.960239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:36.952816 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") pod \"maas-api-e2e-api-a-543d08-6f787759db-nsvhh\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:37.157494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:37.157461 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:37.462571 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:37.462531 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3445476a_8e09_4468_864c_8be7aaa526e9.slice/crio-d7525a2ff3afc26e5662a558ad5e5c50f1b25ce0e9e8c4722ff970f0716ca543 WatchSource:0}: Error finding container d7525a2ff3afc26e5662a558ad5e5c50f1b25ce0e9e8c4722ff970f0716ca543: Status 404 returned error can't find the container with id d7525a2ff3afc26e5662a558ad5e5c50f1b25ce0e9e8c4722ff970f0716ca543 Oct 30 05:08:37.464772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:37.464739 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:08:38.385865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:38.385807 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:08:38.386430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:38.386371 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" containerID="cri-o://e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34" gracePeriod=30 Oct 30 05:08:38.390501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:38.388566 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" event={"ID":"3445476a-8e09-4468-864c-8be7aaa526e9","Type":"ContainerStarted","Data":"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854"} Oct 30 05:08:38.390723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:38.390688 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" event={"ID":"3445476a-8e09-4468-864c-8be7aaa526e9","Type":"ContainerStarted","Data":"d7525a2ff3afc26e5662a558ad5e5c50f1b25ce0e9e8c4722ff970f0716ca543"} Oct 30 05:08:38.390871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:38.390856 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:41.268322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.268213 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" podStartSLOduration=4.738908997 podStartE2EDuration="5.26819301s" totalImagesPullingTime="529.284013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:38.484514672 +0000 UTC m=+1491.454009670" watchObservedRunningTime="2026-10-30 05:08:41.26819301 +0000 UTC m=+1494.237687998" Oct 30 05:08:41.270980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.270952 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:08:41.271819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.271801 5058 cpu_manager.go:414] "containerMap: removing container" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:41.271925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.271822 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451878c2-d0e7-4e41-8cab-80d0bd8f57d1" containerName="istio-proxy" Oct 30 05:08:41.277960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.276502 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.289672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.289505 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:08:41.365142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.365105 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-645aef-serving-cert\"" Oct 30 05:08:41.384894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.384856 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.385653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.385626 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6m8t\" (UniqueName: \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.385837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.385814 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.385966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.385946 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.389300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.389273 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:08:41.395546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.395515 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.419768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.419731 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:08:41.421063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.421036 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c613f5ea\"" Oct 30 05:08:41.449978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.449941 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c613f5ea-dockercfg-m42pw\"" Oct 30 05:08:41.468593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.468557 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:08:41.489718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.489672 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.491887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.491850 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.492018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.491916 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wkf\" (UniqueName: \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.492018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492011 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.492147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492119 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.492281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492188 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzztl\" (UniqueName: \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.492444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492386 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.493519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492458 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.493519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.492580 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6m8t\" (UniqueName: \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.496712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.496684 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.497293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.497271 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.497795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.497770 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:41.501082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.501059 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:08:41.641298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.625835 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wkf\" (UniqueName: \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.641298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.626000 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzztl\" (UniqueName: \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.641298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.626045 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.641298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.626079 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.641298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.627103 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:41.831243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.811894 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.883242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.844423 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzztl\" (UniqueName: \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:41.972140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:41.970687 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:42.013272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.010772 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wkf\" (UniqueName: \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:42.021918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.018933 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:42.021918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.020277 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6m8t\" (UniqueName: \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\") pod \"maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:42.213331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.213021 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:42.346792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.346759 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:08:42.399035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.398983 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:08:42.412688 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:42.412437 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a7da68_6287_46e3_a088_9ff9675c9276.slice/crio-76a395695497856c5cfb9964d696e505cf7d9c433b753cd2b17e8ec21cea2464 WatchSource:0}: Error finding container 76a395695497856c5cfb9964d696e505cf7d9c433b753cd2b17e8ec21cea2464: Status 404 returned error can't find the container with id 76a395695497856c5cfb9964d696e505cf7d9c433b753cd2b17e8ec21cea2464 Oct 30 05:08:42.443420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.443383 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" event={"ID":"11a7da68-6287-46e3-a088-9ff9675c9276","Type":"ContainerStarted","Data":"76a395695497856c5cfb9964d696e505cf7d9c433b753cd2b17e8ec21cea2464"} Oct 30 05:08:42.444803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.444784 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" event={"ID":"728fc409-6170-4b53-9671-8c98543dc33d","Type":"ContainerStarted","Data":"f8d743d9e2c44fd30f31bd616e060d8ba6c6789fe26c5e1af2ccf1499baf54e3"} Oct 30 05:08:42.536988 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:42.536951 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29286179_b14d_4e48_b94d_b5f74c473ff0.slice/crio-741f014c2c0a9d7ea19b398f551881db64104ee8876b681749a35582b933e916 WatchSource:0}: Error finding container 741f014c2c0a9d7ea19b398f551881db64104ee8876b681749a35582b933e916: Status 404 returned error can't find the container with id 741f014c2c0a9d7ea19b398f551881db64104ee8876b681749a35582b933e916 Oct 30 05:08:42.539177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:42.539151 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:08:43.456947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.456904 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" event={"ID":"11a7da68-6287-46e3-a088-9ff9675c9276","Type":"ContainerStarted","Data":"2bdfb5910a6165664509993c0eec27bfb8686b22385ad17f0114844d084abc72"} Oct 30 05:08:43.457449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.456986 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:43.466965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.466934 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" event={"ID":"29286179-b14d-4e48-b94d-b5f74c473ff0","Type":"ContainerStarted","Data":"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a"} Oct 30 05:08:43.467095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.466999 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" event={"ID":"29286179-b14d-4e48-b94d-b5f74c473ff0","Type":"ContainerStarted","Data":"741f014c2c0a9d7ea19b398f551881db64104ee8876b681749a35582b933e916"} Oct 30 05:08:43.467095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.467032 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:43.469393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.469369 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" event={"ID":"728fc409-6170-4b53-9671-8c98543dc33d","Type":"ContainerStarted","Data":"13dae93687929b71d4bd51f1201f0e8045c6d816c75575f19fea7ea9693744eb"} Oct 30 05:08:43.469510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.469495 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:43.501729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.501666 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" podStartSLOduration=2.501648779 podStartE2EDuration="2.501648779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:43.497728081 +0000 UTC m=+1496.467223067" watchObservedRunningTime="2026-10-30 05:08:43.501648779 +0000 UTC m=+1496.471143771" Oct 30 05:08:43.541486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.541425 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" podStartSLOduration=2.541405893 podStartE2EDuration="2.541405893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:43.54020678 +0000 UTC m=+1496.509701770" watchObservedRunningTime="2026-10-30 05:08:43.541405893 +0000 UTC m=+1496.510900880" Oct 30 05:08:43.584898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.584845 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" podStartSLOduration=2.203948814 podStartE2EDuration="2.584828895s" totalImagesPullingTime="380.880081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:43.583225651 +0000 UTC m=+1496.552720630" watchObservedRunningTime="2026-10-30 05:08:43.584828895 +0000 UTC m=+1496.554323883" Oct 30 05:08:43.722436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.720117 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855340 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855513 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855598 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855687 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855748 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855810 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.855891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855863 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.856620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.855958 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.856620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.856009 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\") pod \"322cb168-d8c6-470e-b57c-ca74debc905b\" (UID: \"322cb168-d8c6-470e-b57c-ca74debc905b\") " Oct 30 05:08:43.859650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.859058 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:43.859650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.859405 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:43.860179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.860158 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:43.860524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.860506 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:43.860818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.860801 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:43.882502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.882296 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "kube-api-access-dp25p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:43.882502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.882403 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:43.915176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.915142 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:43.918544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.918494 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy" pod "322cb168-d8c6-470e-b57c-ca74debc905b" (UID: "322cb168-d8c6-470e-b57c-ca74debc905b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:43.958778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958746 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958776 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958786 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958794 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958801 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958809 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cb168-d8c6-470e-b57c-ca74debc905b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958818 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cb168-d8c6-470e-b57c-ca74debc905b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958826 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp25p\" (UniqueName: \"kubernetes.io/projected/322cb168-d8c6-470e-b57c-ca74debc905b-kube-api-access-dp25p\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:43.958982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:43.958835 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cb168-d8c6-470e-b57c-ca74debc905b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:08:44.404010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.403977 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:08:44.482839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.482802 5058 generic.go:356] "Generic (PLEG): container finished" podID="322cb168-d8c6-470e-b57c-ca74debc905b" containerID="e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34" exitCode=0 Oct 30 05:08:44.483422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.482870 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" event={"ID":"322cb168-d8c6-470e-b57c-ca74debc905b","Type":"ContainerDied","Data":"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34"} Oct 30 05:08:44.483422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.482926 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" Oct 30 05:08:44.483422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.482934 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79" event={"ID":"322cb168-d8c6-470e-b57c-ca74debc905b","Type":"ContainerDied","Data":"a15e6fcea3d3d279d6cedf3b9f79f30c851b13eb851e25a3a2403b703665a96c"} Oct 30 05:08:44.483422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.482964 5058 scope.go:122] "RemoveContainer" containerID="e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34" Oct 30 05:08:44.500509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.500479 5058 scope.go:122] "RemoveContainer" containerID="e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34" Oct 30 05:08:44.500888 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:44.500860 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34\": container with ID starting with e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34 not found: ID does not exist" containerID="e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34" Oct 30 05:08:44.501007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.500893 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34"} err="failed to get container status \"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34\": rpc error: code = NotFound desc = could not find container \"e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34\": container with ID starting with e2e7fd3e10d94c215942ea0b248d71259f580ed92231b790ef151ce1e0c13d34 not found: ID does not exist" Oct 30 05:08:44.541728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.541700 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:08:44.545830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:44.545806 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79"] Oct 30 05:08:45.170977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:45.170942 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" path="/var/lib/kubelet/pods/322cb168-d8c6-470e-b57c-ca74debc905b/volumes" Oct 30 05:08:47.476142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:47.476113 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:08:47.494154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:47.493404 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:08:49.191613 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:08:49.191576 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:08:49.488948 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:49.488863 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:08:54.487809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:54.487587 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:08:54.487809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:54.487701 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:08:56.720397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.720358 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:08:56.721713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.721693 5058 cpu_manager.go:414] "containerMap: removing container" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" Oct 30 05:08:56.721713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.721713 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" Oct 30 05:08:56.721855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.721846 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="322cb168-d8c6-470e-b57c-ca74debc905b" containerName="istio-proxy" Oct 30 05:08:56.727670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.727648 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.747400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.747361 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:08:56.755548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.755517 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-288e01-serving-cert\"" Oct 30 05:08:56.823058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.823022 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.823275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.823085 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpzt\" (UniqueName: \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.823275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.823255 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.823601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.823572 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.925107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.925072 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.925299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.925189 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.925299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.925231 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpzt\" (UniqueName: \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.925299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.925277 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.925822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.925797 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.927893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.927863 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.928020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.928003 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:56.945616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:56.945591 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpzt\" (UniqueName: \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\") pod \"maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:57.057662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:57.053869 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:57.505177 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:08:57.505135 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c0fb2_7fb2_4eda_bd4e_e9f849fd99d6.slice/crio-26aa6be22cf482781b9c882a122c2d72705abe7dca8d18de0b3d815e7709e257 WatchSource:0}: Error finding container 26aa6be22cf482781b9c882a122c2d72705abe7dca8d18de0b3d815e7709e257: Status 404 returned error can't find the container with id 26aa6be22cf482781b9c882a122c2d72705abe7dca8d18de0b3d815e7709e257 Oct 30 05:08:57.506316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:57.506293 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:08:57.669765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:57.669724 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" event={"ID":"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6","Type":"ContainerStarted","Data":"26aa6be22cf482781b9c882a122c2d72705abe7dca8d18de0b3d815e7709e257"} Oct 30 05:08:58.679152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:58.679113 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" event={"ID":"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6","Type":"ContainerStarted","Data":"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65"} Oct 30 05:08:58.679564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:58.679228 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:08:58.722535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:08:58.722477 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" podStartSLOduration=2.404559072 podStartE2EDuration="2.722462154s" totalImagesPullingTime="317.903082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:58.722142783 +0000 UTC m=+1511.691637782" watchObservedRunningTime="2026-10-30 05:08:58.722462154 +0000 UTC m=+1511.691957163" Oct 30 05:09:04.695829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:04.695800 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:09:05.645356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.645321 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:09:05.651521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.651498 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.667878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.667853 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:09:05.704482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.704447 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4225df-serving-cert\"" Oct 30 05:09:05.740434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.740399 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.740707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.740679 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.740850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.740833 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.740905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.740863 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767s5\" (UniqueName: \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.842650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.842615 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.842650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.842656 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767s5\" (UniqueName: \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.842902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.842764 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.842902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.842886 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.843488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.843462 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.845133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.845110 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.845273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.845255 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.864314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.864283 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767s5\" (UniqueName: \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\") pod \"maas-api-e2e-inf-b-4225df-b4467595f-ww8cl\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:05.974945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:05.974273 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:06.402883 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:06.402854 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d050b17_db54_4946_a932_734e6b62079b.slice/crio-eb4305a0fa230d20ab028a35975135c13d6b8a4db62194919b2f9bff495f1f50 WatchSource:0}: Error finding container eb4305a0fa230d20ab028a35975135c13d6b8a4db62194919b2f9bff495f1f50: Status 404 returned error can't find the container with id eb4305a0fa230d20ab028a35975135c13d6b8a4db62194919b2f9bff495f1f50 Oct 30 05:09:06.404735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:06.404709 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:09:06.748607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:06.748528 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" event={"ID":"3d050b17-db54-4946-a932-734e6b62079b","Type":"ContainerStarted","Data":"eb4305a0fa230d20ab028a35975135c13d6b8a4db62194919b2f9bff495f1f50"} Oct 30 05:09:07.757310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:07.757272 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" event={"ID":"3d050b17-db54-4946-a932-734e6b62079b","Type":"ContainerStarted","Data":"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae"} Oct 30 05:09:07.757877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:07.757846 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:07.802605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:07.802553 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" podStartSLOduration=2.452860282 podStartE2EDuration="2.802539566s" totalImagesPullingTime="349.679284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:07.79954852 +0000 UTC m=+1520.769043500" watchObservedRunningTime="2026-10-30 05:09:07.802539566 +0000 UTC m=+1520.772034553" Oct 30 05:09:08.784473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:08.784442 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:09:08.784890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:08.784648 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" containerID="cri-o://a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001" gracePeriod=30 Oct 30 05:09:08.799474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:08.799439 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:09:08.799652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:08.799631 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" containerID="cri-o://ee763c4e586ae9769f82f14998a307fdbc80739a9fdbedea45bcd2c719290211" gracePeriod=30 Oct 30 05:09:09.544458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.544425 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:09:09.544689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.544663 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" containerID="cri-o://9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67" gracePeriod=30 Oct 30 05:09:09.557301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.557262 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:09:09.558162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.558104 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" containerID="cri-o://b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da" gracePeriod=30 Oct 30 05:09:09.563203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.563170 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 05:09:09.563492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.563443 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" containerID="cri-o://3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861" gracePeriod=30 Oct 30 05:09:09.573134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.573103 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 05:09:09.573400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:09.573375 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" containerID="cri-o://c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7" gracePeriod=30 Oct 30 05:09:13.044963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.044930 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:09:13.052869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.051490 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.068460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.068244 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:09:13.075812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.075781 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c613f5ea-serving-cert\"" Oct 30 05:09:13.112443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.112410 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2td\" (UniqueName: \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.112640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.112477 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.112640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.112563 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.112809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.112736 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.214102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.214067 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.214331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.214314 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2td\" (UniqueName: \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.214512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.214408 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.214512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.214457 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.214897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.214869 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.216955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.216929 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.217043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.217025 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.269242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.269181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2td\" (UniqueName: \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\") pod \"maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.389759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.389573 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:13.732195 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:13.732162 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25f7963_5f1c_423b_a63c_6aee2121faad.slice/crio-90125c80c04ee15259dfc8363eefa08a7179d9d681cef571f08008762ecbe988 WatchSource:0}: Error finding container 90125c80c04ee15259dfc8363eefa08a7179d9d681cef571f08008762ecbe988: Status 404 returned error can't find the container with id 90125c80c04ee15259dfc8363eefa08a7179d9d681cef571f08008762ecbe988 Oct 30 05:09:13.734064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.734038 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:09:13.771736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.771710 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:09:13.836897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:13.836854 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" event={"ID":"c25f7963-5f1c-423b-a63c-6aee2121faad","Type":"ContainerStarted","Data":"90125c80c04ee15259dfc8363eefa08a7179d9d681cef571f08008762ecbe988"} Oct 30 05:09:14.195596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.195573 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:09:14.375736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.375702 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.376023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.375991 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.376520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.376485 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.376758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.376741 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.377017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.377002 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.377528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.377505 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.393240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.388550 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.393714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.393690 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.394787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.394765 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\") pod \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\" (UID: \"66b6dc71-f1f7-46d3-bc0f-d43fa658c968\") " Oct 30 05:09:14.399053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.399027 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:14.399946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.399825 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:14.400058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.400003 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:14.404065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.403968 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:14.493212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.460693 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:14.533070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.522122 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.533070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.522158 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.533070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.522172 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.533070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.522184 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.533070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.522197 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.722291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.721683 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "kube-api-access-dkkrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:14.742527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.730399 5058 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:55294->10.132.0.64:15021: read: connection reset by peer" start-of-body= Oct 30 05:09:14.742527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.730461 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:55294->10.132.0.64:15021: read: connection reset by peer" Oct 30 05:09:14.830536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.747833 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkrq\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-kube-api-access-dkkrq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.835786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.835706 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:14.836144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.836101 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.859315 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.859350 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.880473 5058 generic.go:356] "Generic (PLEG): container finished" podID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerID="a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001" exitCode=0 Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.880596 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" event={"ID":"66b6dc71-f1f7-46d3-bc0f-d43fa658c968","Type":"ContainerDied","Data":"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001"} Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.880645 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" event={"ID":"66b6dc71-f1f7-46d3-bc0f-d43fa658c968","Type":"ContainerDied","Data":"e08f52817a7b82fc6a5ebe9aa3c0abd9c76c4b2b4f3c84ff4ceecbc5a50d8de6"} Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.880673 5058 scope.go:122] "RemoveContainer" containerID="a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001" Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.880834 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp" Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.884784 5058 generic.go:356] "Generic (PLEG): container finished" podID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerID="ee763c4e586ae9769f82f14998a307fdbc80739a9fdbedea45bcd2c719290211" exitCode=0 Oct 30 05:09:14.888293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.884922 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" event={"ID":"fb51a19b-b580-42cb-a7fe-30ae85662bed","Type":"ContainerDied","Data":"ee763c4e586ae9769f82f14998a307fdbc80739a9fdbedea45bcd2c719290211"} Oct 30 05:09:14.901727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.901560 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo" pod "66b6dc71-f1f7-46d3-bc0f-d43fa658c968" (UID: "66b6dc71-f1f7-46d3-bc0f-d43fa658c968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:14.909879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.909422 5058 scope.go:122] "RemoveContainer" containerID="a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001" Oct 30 05:09:14.909879 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:14.909748 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001\": container with ID starting with a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001 not found: ID does not exist" containerID="a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001" Oct 30 05:09:14.909879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.909790 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001"} err="failed to get container status \"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001\": rpc error: code = NotFound desc = could not find container \"a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001\": container with ID starting with a2b3db41d5ee84317a67c195bb7979a0340c467c548fc46729d777681b657001 not found: ID does not exist" Oct 30 05:09:14.945879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.945415 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:09:14.962262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:14.962191 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6dc71-f1f7-46d3-bc0f-d43fa658c968-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078197 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078297 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078332 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078440 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078495 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078600 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078641 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078718 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.079877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.078800 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\") pod \"fb51a19b-b580-42cb-a7fe-30ae85662bed\" (UID: \"fb51a19b-b580-42cb-a7fe-30ae85662bed\") " Oct 30 05:09:15.080793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.080639 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.081054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.080904 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.082021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.081254 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:15.082737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.082595 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.082737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.082655 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.090265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.090196 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.091285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.091191 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.091499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.091480 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:15.100341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.100267 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr" pod "fb51a19b-b580-42cb-a7fe-30ae85662bed" (UID: "fb51a19b-b580-42cb-a7fe-30ae85662bed"). InnerVolumeSpecName "kube-api-access-qgnnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.113876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.113455 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 05:09:15.287666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.287624 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.288444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.288424 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.288658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.288628 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.288833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.288807 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.289836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.289815 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.290487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.289296 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:15.299028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.297677 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.300182 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.300253 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.300287 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.300627 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\") pod \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\" (UID: \"de7049d7-cd5e-4b6b-bd9e-17753d20db18\") " Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301742 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301766 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301782 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301795 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301810 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301824 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301838 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301850 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb51a19b-b580-42cb-a7fe-30ae85662bed-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301863 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301876 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb51a19b-b580-42cb-a7fe-30ae85662bed-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.301889 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnnr\" (UniqueName: \"kubernetes.io/projected/fb51a19b-b580-42cb-a7fe-30ae85662bed-kube-api-access-qgnnr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.302374 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.302565 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.302583 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.304532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.302856 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.333893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.333842 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.334561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.334529 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:15.339843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.339811 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx" pod "de7049d7-cd5e-4b6b-bd9e-17753d20db18" (UID: "de7049d7-cd5e-4b6b-bd9e-17753d20db18"). InnerVolumeSpecName "kube-api-access-4fnbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.403454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403425 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnbx\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-kube-api-access-4fnbx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403451 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403460 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403468 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403477 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403484 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.403623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.403492 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7049d7-cd5e-4b6b-bd9e-17753d20db18-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.404614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.404589 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:09:15.421513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.421487 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:09:15.504195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504169 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504270 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504298 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504383 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504401 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504462 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504514 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504571 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504617 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504661 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504712 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504768 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504806 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.504997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.504854 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.505835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.505246 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.505835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.505645 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.505835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.505731 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.505984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.505852 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.505984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.505955 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.506417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506391 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.506573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506472 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.506573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506543 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.506691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506577 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\") pod \"123f9d88-19c1-48e0-b756-5d369cf0ea44\" (UID: \"123f9d88-19c1-48e0-b756-5d369cf0ea44\") " Oct 30 05:09:15.506691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506662 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\") pod \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\" (UID: \"3be6a7f3-6203-4b37-ac7c-4d330075ebbd\") " Oct 30 05:09:15.506917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.506897 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:15.507192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.507172 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:15.507443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.507426 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.508324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.508015 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.508997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.508971 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.509301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509279 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509859 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509884 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509898 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f9d88-19c1-48e0-b756-5d369cf0ea44-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509910 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509923 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509936 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509950 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509962 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509974 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509985 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.509997 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.510031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.510008 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.511036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.511012 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.511036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.511048 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "kube-api-access-5vrd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.511292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.511067 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "kube-api-access-cx8sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.511503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.511477 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy" pod "123f9d88-19c1-48e0-b756-5d369cf0ea44" (UID: "123f9d88-19c1-48e0-b756-5d369cf0ea44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.511859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.511606 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:15.512970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.512947 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy" pod "3be6a7f3-6203-4b37-ac7c-4d330075ebbd" (UID: "3be6a7f3-6203-4b37-ac7c-4d330075ebbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.517978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.517952 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 05:09:15.555328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.555292 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:09:15.562659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.562626 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp"] Oct 30 05:09:15.610994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.610965 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.610994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611001 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611045 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611099 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611185 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611261 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611417 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611436 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611468 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611499 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611500 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.611629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611627 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.611991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.611683 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\") pod \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\" (UID: \"e4be6d48-39d7-4b6c-9757-92bfa08c02e9\") " Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612110 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612480 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612501 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612513 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f9d88-19c1-48e0-b756-5d369cf0ea44-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612526 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612538 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612549 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612561 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612574 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612587 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612600 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrd8\" (UniqueName: \"kubernetes.io/projected/3be6a7f3-6203-4b37-ac7c-4d330075ebbd-kube-api-access-5vrd8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.612637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.612613 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8sr\" (UniqueName: \"kubernetes.io/projected/123f9d88-19c1-48e0-b756-5d369cf0ea44-kube-api-access-cx8sr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.613778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.613752 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:15.613953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.613933 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "kube-api-access-ppr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.614082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.614065 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:15.614963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.614942 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo" pod "e4be6d48-39d7-4b6c-9757-92bfa08c02e9" (UID: "e4be6d48-39d7-4b6c-9757-92bfa08c02e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:15.714742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.714699 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.714742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.714741 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.715028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.714757 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppr2m\" (UniqueName: \"kubernetes.io/projected/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-kube-api-access-ppr2m\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.715028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.714771 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4be6d48-39d7-4b6c-9757-92bfa08c02e9-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:15.930479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.928377 5058 generic.go:356] "Generic (PLEG): container finished" podID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerID="3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861" exitCode=0 Oct 30 05:09:15.930479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.928467 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" event={"ID":"e4be6d48-39d7-4b6c-9757-92bfa08c02e9","Type":"ContainerDied","Data":"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861"} Oct 30 05:09:15.930479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.928517 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" event={"ID":"e4be6d48-39d7-4b6c-9757-92bfa08c02e9","Type":"ContainerDied","Data":"160f2e848b11b6cc28de11ecf9b172845855c52e9fa6cdc52ab8aa54dfe6757b"} Oct 30 05:09:15.930479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.928548 5058 scope.go:122] "RemoveContainer" containerID="3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861" Oct 30 05:09:15.930479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.928664 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw" Oct 30 05:09:15.944440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.944415 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" Oct 30 05:09:15.944564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.944469 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v" event={"ID":"fb51a19b-b580-42cb-a7fe-30ae85662bed","Type":"ContainerDied","Data":"55167e753112a5cc6488b493e224d06e75322cbed2a46879255a048b6f32844a"} Oct 30 05:09:15.947360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.946983 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" event={"ID":"c25f7963-5f1c-423b-a63c-6aee2121faad","Type":"ContainerStarted","Data":"6f23d584f01e72314a542a4d6064bf5f0c65c874120fc2eb3371af9cf8c35e78"} Oct 30 05:09:15.965415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.948144 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:15.985243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.974905 5058 generic.go:356] "Generic (PLEG): container finished" podID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerID="9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67" exitCode=0 Oct 30 05:09:15.985243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.975123 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" event={"ID":"123f9d88-19c1-48e0-b756-5d369cf0ea44","Type":"ContainerDied","Data":"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67"} Oct 30 05:09:15.985243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.975172 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" event={"ID":"123f9d88-19c1-48e0-b756-5d369cf0ea44","Type":"ContainerDied","Data":"ed95c35ec69e491bd2782eb1c4d73fe0505456e638aa25fabb7805e96482b81e"} Oct 30 05:09:15.985243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:15.975265 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4" Oct 30 05:09:16.053315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.047465 5058 scope.go:122] "RemoveContainer" containerID="3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861" Oct 30 05:09:16.086054 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:16.054660 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861\": container with ID starting with 3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861 not found: ID does not exist" containerID="3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861" Oct 30 05:09:16.086260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.086119 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861"} err="failed to get container status \"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861\": rpc error: code = NotFound desc = could not find container \"3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861\": container with ID starting with 3d4494361f78f9000cd2c25abdd501f539c4ae9f0ca047e31dc467f29c3ff861 not found: ID does not exist" Oct 30 05:09:16.086260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.086156 5058 scope.go:122] "RemoveContainer" containerID="ee763c4e586ae9769f82f14998a307fdbc80739a9fdbedea45bcd2c719290211" Oct 30 05:09:16.086402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.074253 5058 generic.go:356] "Generic (PLEG): container finished" podID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerID="b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da" exitCode=0 Oct 30 05:09:16.086402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.074283 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" event={"ID":"3be6a7f3-6203-4b37-ac7c-4d330075ebbd","Type":"ContainerDied","Data":"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da"} Oct 30 05:09:16.086514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.086427 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:09:16.086514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.086455 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" event={"ID":"3be6a7f3-6203-4b37-ac7c-4d330075ebbd","Type":"ContainerDied","Data":"1ffd1702da735860c682f0d73afc77101fb8867a2beda8a4e828504f592c2b4b"} Oct 30 05:09:16.086514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.074383 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6" Oct 30 05:09:16.115459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.087814 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v"] Oct 30 05:09:16.160430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.160388 5058 scope.go:122] "RemoveContainer" containerID="9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67" Oct 30 05:09:16.170646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.160952 5058 generic.go:356] "Generic (PLEG): container finished" podID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerID="c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7" exitCode=0 Oct 30 05:09:16.171314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.171276 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" event={"ID":"de7049d7-cd5e-4b6b-bd9e-17753d20db18","Type":"ContainerDied","Data":"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7"} Oct 30 05:09:16.171513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.171496 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" event={"ID":"de7049d7-cd5e-4b6b-bd9e-17753d20db18","Type":"ContainerDied","Data":"7a25fc7dc79835cbeb4ef12275d8b6b6e42ec0a645ce40bda734192ab5e7934f"} Oct 30 05:09:16.175418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.175390 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n" Oct 30 05:09:16.227041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.227002 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 05:09:16.298427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.298385 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw"] Oct 30 05:09:16.300178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.300113 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" podStartSLOduration=2.843809614 podStartE2EDuration="3.300094479s" totalImagesPullingTime="456.284865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:16.194057666 +0000 UTC m=+1529.163552651" watchObservedRunningTime="2026-10-30 05:09:16.300094479 +0000 UTC m=+1529.269589483" Oct 30 05:09:16.433139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.433095 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:09:16.434383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.434363 5058 scope.go:122] "RemoveContainer" containerID="9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67" Oct 30 05:09:16.435155 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:16.435101 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67\": container with ID starting with 9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67 not found: ID does not exist" containerID="9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67" Oct 30 05:09:16.435401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.435329 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67"} err="failed to get container status \"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67\": rpc error: code = NotFound desc = could not find container \"9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67\": container with ID starting with 9f815bbd34f2e1b12b608e544d3c1e68589d260259a70f6a01aa7f678fa37b67 not found: ID does not exist" Oct 30 05:09:16.435401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.435363 5058 scope.go:122] "RemoveContainer" containerID="b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da" Oct 30 05:09:16.440688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.440633 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4"] Oct 30 05:09:16.463333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.463305 5058 scope.go:122] "RemoveContainer" containerID="b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da" Oct 30 05:09:16.463855 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:16.463828 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da\": container with ID starting with b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da not found: ID does not exist" containerID="b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da" Oct 30 05:09:16.463961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.463863 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da"} err="failed to get container status \"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da\": rpc error: code = NotFound desc = could not find container \"b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da\": container with ID starting with b07a544e741b05820a772923f3faccc127966f3cf9083b2df94a2ee82f6dd7da not found: ID does not exist" Oct 30 05:09:16.463961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.463881 5058 scope.go:122] "RemoveContainer" containerID="c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7" Oct 30 05:09:16.464673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.464650 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 05:09:16.469069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.469048 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n"] Oct 30 05:09:16.475092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.475075 5058 scope.go:122] "RemoveContainer" containerID="c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7" Oct 30 05:09:16.475417 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:16.475399 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7\": container with ID starting with c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7 not found: ID does not exist" containerID="c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7" Oct 30 05:09:16.475486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.475427 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7"} err="failed to get container status \"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7\": rpc error: code = NotFound desc = could not find container \"c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7\": container with ID starting with c034a5fbde928035d998ac5f72659fdadc386dbf73e025be57d510039ce724d7 not found: ID does not exist" Oct 30 05:09:16.500550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.500521 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:09:16.504200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:16.504179 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6"] Oct 30 05:09:17.169370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.169329 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" path="/var/lib/kubelet/pods/123f9d88-19c1-48e0-b756-5d369cf0ea44/volumes" Oct 30 05:09:17.170164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.170137 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" path="/var/lib/kubelet/pods/3be6a7f3-6203-4b37-ac7c-4d330075ebbd/volumes" Oct 30 05:09:17.170876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.170850 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" path="/var/lib/kubelet/pods/66b6dc71-f1f7-46d3-bc0f-d43fa658c968/volumes" Oct 30 05:09:17.171985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.171960 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" path="/var/lib/kubelet/pods/de7049d7-cd5e-4b6b-bd9e-17753d20db18/volumes" Oct 30 05:09:17.172673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.172652 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" path="/var/lib/kubelet/pods/e4be6d48-39d7-4b6c-9757-92bfa08c02e9/volumes" Oct 30 05:09:17.173346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:17.173319 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" path="/var/lib/kubelet/pods/fb51a19b-b580-42cb-a7fe-30ae85662bed/volumes" Oct 30 05:09:19.305424 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:19.303183 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:09:23.215359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:23.215325 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:09:27.180293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.178609 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:09:27.180293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.179983 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" Oct 30 05:09:27.180293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180253 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" Oct 30 05:09:27.180293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180280 5058 cpu_manager.go:414] "containerMap: removing container" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180311 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180338 5058 cpu_manager.go:414] "containerMap: removing container" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180346 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180384 5058 cpu_manager.go:414] "containerMap: removing container" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180392 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180407 5058 cpu_manager.go:414] "containerMap: removing container" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180415 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180443 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180451 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.180990 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4be6d48-39d7-4b6c-9757-92bfa08c02e9" containerName="istio-proxy" Oct 30 05:09:27.181022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.181022 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be6a7f3-6203-4b37-ac7c-4d330075ebbd" containerName="istio-proxy" Oct 30 05:09:27.181526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.181032 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="123f9d88-19c1-48e0-b756-5d369cf0ea44" containerName="istio-proxy" Oct 30 05:09:27.181526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.181047 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b6dc71-f1f7-46d3-bc0f-d43fa658c968" containerName="istio-proxy" Oct 30 05:09:27.181526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.181060 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7049d7-cd5e-4b6b-bd9e-17753d20db18" containerName="istio-proxy" Oct 30 05:09:27.181526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.181069 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb51a19b-b580-42cb-a7fe-30ae85662bed" containerName="istio-proxy" Oct 30 05:09:27.200861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.200812 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.216362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.216251 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:09:27.217645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.217433 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3d55bd-openshift-default-dockercfg-65sqc\"" Oct 30 05:09:27.226838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.226807 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.226988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.226858 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.226988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.226895 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.226983 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.227025 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52r2\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.227105 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.227120 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.227147 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.227344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.227178 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.329832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.329795 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.329832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.329839 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.329863 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.329893 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.329921 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52r2\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330000 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330017 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330051 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330094 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330395 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330448 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330523 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.330744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.330721 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.332315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.332294 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.332560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.332543 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.350903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.350872 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.351097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.351072 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52r2\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\") pod \"e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:27.522969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:27.522874 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:28.014488 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:28.014450 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d2633a_d368_4dbb_8e6d_e8808a45fcad.slice/crio-f94b9ee9f73cc287e58feafa0303547484a43fe6d731d16d3c04fd091b0b2751 WatchSource:0}: Error finding container f94b9ee9f73cc287e58feafa0303547484a43fe6d731d16d3c04fd091b0b2751: Status 404 returned error can't find the container with id f94b9ee9f73cc287e58feafa0303547484a43fe6d731d16d3c04fd091b0b2751 Oct 30 05:09:28.016623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.016601 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:09:28.017542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.017515 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:28.017596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.017580 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:28.017627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.017609 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:28.460469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.460439 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerStarted","Data":"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5"} Oct 30 05:09:28.461100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.461066 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerStarted","Data":"f94b9ee9f73cc287e58feafa0303547484a43fe6d731d16d3c04fd091b0b2751"} Oct 30 05:09:28.524371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.524342 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:28.540203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:28.540153 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podStartSLOduration=1.5401391229999999 podStartE2EDuration="1.540139123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:28.526487514 +0000 UTC m=+1541.495982501" watchObservedRunningTime="2026-10-30 05:09:28.540139123 +0000 UTC m=+1541.509634110" Oct 30 05:09:29.530603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:29.530567 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:30.508244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:30.498437 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:30.508244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:30.500373 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:09:32.824862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:32.824825 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:09:32.831016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:32.830991 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:32.848936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:32.848906 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:09:32.852676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:32.852652 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d55bd\"" Oct 30 05:09:32.970244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:32.932584 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3d55bd-dockercfg-hlxbp\"" Oct 30 05:09:33.027243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.026544 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.027243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.026597 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkjl\" (UniqueName: \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.133319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.131103 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.133319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.131157 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkjl\" (UniqueName: \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.133319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.132137 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.186241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.178096 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkjl\" (UniqueName: \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\") pod \"payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.452490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.452455 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:33.756598 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:33.756568 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7808cbe3_5187_4e5f_9a32_d233eff308fb.slice/crio-2d6aaac3a00ac4367d5ec9480fd8d68acadc36518e17014b24fbc8081fa8c7ea WatchSource:0}: Error finding container 2d6aaac3a00ac4367d5ec9480fd8d68acadc36518e17014b24fbc8081fa8c7ea: Status 404 returned error can't find the container with id 2d6aaac3a00ac4367d5ec9480fd8d68acadc36518e17014b24fbc8081fa8c7ea Oct 30 05:09:33.758164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:33.758140 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:09:34.532832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:34.532794 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" event={"ID":"7808cbe3-5187-4e5f-9a32-d233eff308fb","Type":"ContainerStarted","Data":"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2"} Oct 30 05:09:34.533325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:34.532852 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" event={"ID":"7808cbe3-5187-4e5f-9a32-d233eff308fb","Type":"ContainerStarted","Data":"2d6aaac3a00ac4367d5ec9480fd8d68acadc36518e17014b24fbc8081fa8c7ea"} Oct 30 05:09:34.533325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:34.532895 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:34.579582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:34.579512 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" podStartSLOduration=2.579495857 podStartE2EDuration="2.579495857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:34.573241693 +0000 UTC m=+1547.542736679" watchObservedRunningTime="2026-10-30 05:09:34.579495857 +0000 UTC m=+1547.548990850" Oct 30 05:09:39.215992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.215954 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 05:09:39.216974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.216251 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" containerID="cri-o://81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc" gracePeriod=30 Oct 30 05:09:39.234363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.234329 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 05:09:39.234815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.234785 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" containerID="cri-o://c46114eb6d71ab11131c96826ba6763a79e881095092554e3a5c28eef79676e1" gracePeriod=30 Oct 30 05:09:39.242907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.242876 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:09:39.243501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.243299 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" containerID="cri-o://7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f" gracePeriod=30 Oct 30 05:09:39.248003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.247973 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:09:39.248333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:39.248209 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" containerID="cri-o://d25723c7cef81106d25494d08c2d9eb1f694a97f425923cd28eaa8fc2a7eacb7" gracePeriod=30 Oct 30 05:09:44.571179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.570763 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 05:09:44.621350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.621288 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:09:44.674734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.673970 5058 generic.go:356] "Generic (PLEG): container finished" podID="6183adbe-4377-4a22-a1ee-a205d5951402" containerID="7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f" exitCode=0 Oct 30 05:09:44.674734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.674282 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" event={"ID":"6183adbe-4377-4a22-a1ee-a205d5951402","Type":"ContainerDied","Data":"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f"} Oct 30 05:09:44.674734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.674345 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" event={"ID":"6183adbe-4377-4a22-a1ee-a205d5951402","Type":"ContainerDied","Data":"49b169139a830239112864fe037f058779f8d38aaee3d55e6274634d43b70abc"} Oct 30 05:09:44.674734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.674375 5058 scope.go:122] "RemoveContainer" containerID="7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f" Oct 30 05:09:44.674734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.674542 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx" Oct 30 05:09:44.678746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.678697 5058 generic.go:356] "Generic (PLEG): container finished" podID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerID="81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc" exitCode=0 Oct 30 05:09:44.678892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.678766 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" Oct 30 05:09:44.679050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.679019 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" event={"ID":"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6","Type":"ContainerDied","Data":"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc"} Oct 30 05:09:44.679125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.679073 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv" event={"ID":"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6","Type":"ContainerDied","Data":"1ee3a48ef1b5f00191230318043c892573a2e306b380aed2e931b1c4765a6dc2"} Oct 30 05:09:44.684377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.684113 5058 generic.go:356] "Generic (PLEG): container finished" podID="9c62b671-8729-4713-b94c-1c02da5e1388" containerID="c46114eb6d71ab11131c96826ba6763a79e881095092554e3a5c28eef79676e1" exitCode=0 Oct 30 05:09:44.684377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.684229 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerDied","Data":"c46114eb6d71ab11131c96826ba6763a79e881095092554e3a5c28eef79676e1"} Oct 30 05:09:44.688004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.687930 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" event={"ID":"9dd5a154-07d7-46f3-a114-6c8904afbf51","Type":"ContainerDied","Data":"d25723c7cef81106d25494d08c2d9eb1f694a97f425923cd28eaa8fc2a7eacb7"} Oct 30 05:09:44.688207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.687803 5058 generic.go:356] "Generic (PLEG): container finished" podID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerID="d25723c7cef81106d25494d08c2d9eb1f694a97f425923cd28eaa8fc2a7eacb7" exitCode=0 Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728195 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728282 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728316 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728375 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728403 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728453 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728560 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728616 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728650 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728707 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728742 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728770 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728810 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728845 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728919 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.728975 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\") pod \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\" (UID: \"05f2ade0-ac3a-41bd-a4b8-867656f8e9e6\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.729061 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.733892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.729086 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\") pod \"6183adbe-4377-4a22-a1ee-a205d5951402\" (UID: \"6183adbe-4377-4a22-a1ee-a205d5951402\") " Oct 30 05:09:44.745200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.744706 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.745200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.744941 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.745828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.745685 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.746106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.745952 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.762346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.759933 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.765330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.764337 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.765330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.764599 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:44.765330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.765244 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "kube-api-access-8jbsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.781683 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.782009 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.786253 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.786809 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.786988 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.787767 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.792037 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.793468 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.794995 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v" pod "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" (UID: "05f2ade0-ac3a-41bd-a4b8-867656f8e9e6"). InnerVolumeSpecName "kube-api-access-qnx5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.796464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.796440 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy" pod "6183adbe-4377-4a22-a1ee-a205d5951402" (UID: "6183adbe-4377-4a22-a1ee-a205d5951402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.801356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.801267 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:09:44.811768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.810885 5058 scope.go:122] "RemoveContainer" containerID="7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f" Oct 30 05:09:44.811768 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:44.811282 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f\": container with ID starting with 7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f not found: ID does not exist" containerID="7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f" Oct 30 05:09:44.811768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.811320 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f"} err="failed to get container status \"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f\": rpc error: code = NotFound desc = could not find container \"7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f\": container with ID starting with 7a335447b5b8a373aecc3d04691e2453c8b4d0d67c88a26882d8f53d5a6de06f not found: ID does not exist" Oct 30 05:09:44.811768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.811343 5058 scope.go:122] "RemoveContainer" containerID="81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc" Oct 30 05:09:44.827674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.825111 5058 scope.go:122] "RemoveContainer" containerID="81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc" Oct 30 05:09:44.827674 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:44.825437 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc\": container with ID starting with 81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc not found: ID does not exist" containerID="81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc" Oct 30 05:09:44.827674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.825475 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc"} err="failed to get container status \"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc\": rpc error: code = NotFound desc = could not find container \"81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc\": container with ID starting with 81cac8ed22083a479005aaf9a54fb05c15b43547f1cdb286d9b33dfe7b66b0fc not found: ID does not exist" Oct 30 05:09:44.827674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.825499 5058 scope.go:122] "RemoveContainer" containerID="b1e6adbcfc8492f233c58d1241871bd062d656c42c268a27f5230867d1a83dab" Oct 30 05:09:44.830623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.830596 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 05:09:44.833587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833560 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6183adbe-4377-4a22-a1ee-a205d5951402-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833587 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833601 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833613 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6183adbe-4377-4a22-a1ee-a205d5951402-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833626 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833637 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833657 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833670 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833681 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833692 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnx5v\" (UniqueName: \"kubernetes.io/projected/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-kube-api-access-qnx5v\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833705 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jbsj\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-kube-api-access-8jbsj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833718 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833730 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833742 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833754 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833766 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.833774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833778 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6183adbe-4377-4a22-a1ee-a205d5951402-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.834242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.833793 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6183adbe-4377-4a22-a1ee-a205d5951402-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.842142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.842113 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:09:44.843016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.842994 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" Oct 30 05:09:44.843016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843012 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843025 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843030 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843045 5058 cpu_manager.go:414] "containerMap: removing container" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843050 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843076 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" Oct 30 05:09:44.843166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843081 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" Oct 30 05:09:44.844023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843281 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" containerName="istio-proxy" Oct 30 05:09:44.844023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843303 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" containerName="istio-proxy" Oct 30 05:09:44.844023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843316 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:44.844023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843328 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" containerName="istio-proxy" Oct 30 05:09:44.844023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.843341 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:44.848746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.848723 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.858373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.858347 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:09:44.865303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.865277 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-47af37-openshift-default-dockercfg-xfsdb\"" Oct 30 05:09:44.935510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935474 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.935687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935528 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.935687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935584 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.935687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935635 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935742 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935777 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935811 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935886 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.935958 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936037 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936072 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936123 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936191 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936244 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936279 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936306 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936382 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\") pod \"9c62b671-8729-4713-b94c-1c02da5e1388\" (UID: \"9c62b671-8729-4713-b94c-1c02da5e1388\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936453 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\") pod \"9dd5a154-07d7-46f3-a114-6c8904afbf51\" (UID: \"9dd5a154-07d7-46f3-a114-6c8904afbf51\") " Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936665 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936725 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936790 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936813 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.936989 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zbr\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.937051 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.937206 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.937277 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.937333 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:44.939495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.937750 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.942300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.942272 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.942973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.942625 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.942973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.942661 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.942973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.942699 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.942973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.942963 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:44.943820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.943555 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:44.944355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.944312 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.944505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.944484 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.944795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.944776 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.945541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.945514 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.949544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949518 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.949675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949544 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:44.949675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949577 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "kube-api-access-cjtxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.949675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949599 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.949841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949613 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token" pod "9c62b671-8729-4713-b94c-1c02da5e1388" (UID: "9c62b671-8729-4713-b94c-1c02da5e1388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.949841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.949621 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:44.950888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:44.950866 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt" pod "9dd5a154-07d7-46f3-a114-6c8904afbf51" (UID: "9dd5a154-07d7-46f3-a114-6c8904afbf51"). InnerVolumeSpecName "kube-api-access-klmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:45.028751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.028717 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:09:45.038396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038362 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038415 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038455 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038539 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038630 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038657 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038786 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zbr\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.038956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038840 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.039013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038955 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038973 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.038988 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c62b671-8729-4713-b94c-1c02da5e1388-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039002 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c62b671-8729-4713-b94c-1c02da5e1388-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039017 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039030 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039043 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039057 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmqt\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-kube-api-access-klmqt\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039071 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039084 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039098 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039112 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039124 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039137 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjtxw\" (UniqueName: \"kubernetes.io/projected/9c62b671-8729-4713-b94c-1c02da5e1388-kube-api-access-cjtxw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039151 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd5a154-07d7-46f3-a114-6c8904afbf51-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039166 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c62b671-8729-4713-b94c-1c02da5e1388-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039177 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd5a154-07d7-46f3-a114-6c8904afbf51-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.039292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039189 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd5a154-07d7-46f3-a114-6c8904afbf51-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.040000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039314 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.040000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039499 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.040000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.039741 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.040341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.040311 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx"] Oct 30 05:09:45.040435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.040355 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.040937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.040915 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.042560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.042534 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.042655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.042603 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.058984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.058920 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.060046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.060027 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zbr\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\") pod \"e2e-api-b-47af37-openshift-default-677f546d75-h4nn4\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.074995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.074729 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 05:09:45.131241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.097706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv"] Oct 30 05:09:45.211442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.206250 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:45.435865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.435829 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f2ade0-ac3a-41bd-a4b8-867656f8e9e6" path="/var/lib/kubelet/pods/05f2ade0-ac3a-41bd-a4b8-867656f8e9e6/volumes" Oct 30 05:09:45.436982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.436959 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6183adbe-4377-4a22-a1ee-a205d5951402" path="/var/lib/kubelet/pods/6183adbe-4377-4a22-a1ee-a205d5951402/volumes" Oct 30 05:09:45.543149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.543123 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:09:45.713024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.712995 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" Oct 30 05:09:45.713517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.713001 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4" event={"ID":"9c62b671-8729-4713-b94c-1c02da5e1388","Type":"ContainerDied","Data":"df5d09b7f5a1b94614e50d2e4efa2b1d4b7ec3ff161086e8d0632b79d51ed531"} Oct 30 05:09:45.713517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.713157 5058 scope.go:122] "RemoveContainer" containerID="c46114eb6d71ab11131c96826ba6763a79e881095092554e3a5c28eef79676e1" Oct 30 05:09:45.714691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.714655 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" event={"ID":"9dd5a154-07d7-46f3-a114-6c8904afbf51","Type":"ContainerDied","Data":"871fa8008afdf724e493cea04350c79560e98c2c58f7ad1d19dcfbcdba81ce1f"} Oct 30 05:09:45.714800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.714745 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m" Oct 30 05:09:45.724085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.724066 5058 scope.go:122] "RemoveContainer" containerID="d25723c7cef81106d25494d08c2d9eb1f694a97f425923cd28eaa8fc2a7eacb7" Oct 30 05:09:45.771691 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:45.771662 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fdebf5_5fda_4d80_82f8_06b1a7bb5f2a.slice/crio-756cea45945790762a4c033f3fd53741370a0253d03c2056985918048b8adff2 WatchSource:0}: Error finding container 756cea45945790762a4c033f3fd53741370a0253d03c2056985918048b8adff2: Status 404 returned error can't find the container with id 756cea45945790762a4c033f3fd53741370a0253d03c2056985918048b8adff2 Oct 30 05:09:45.774934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.774902 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:45.775022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.774975 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:45.775022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.775008 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:45.781644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.781619 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:09:45.809639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.809610 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 05:09:45.815717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.815691 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4"] Oct 30 05:09:45.846707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.846675 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:09:45.852607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:45.852583 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m"] Oct 30 05:09:46.742921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:46.733450 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerStarted","Data":"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57"} Oct 30 05:09:46.742921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:46.733534 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerStarted","Data":"756cea45945790762a4c033f3fd53741370a0253d03c2056985918048b8adff2"} Oct 30 05:09:46.801242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:46.798112 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podStartSLOduration=2.798088356 podStartE2EDuration="2.798088356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:46.791832383 +0000 UTC m=+1559.761327371" watchObservedRunningTime="2026-10-30 05:09:46.798088356 +0000 UTC m=+1559.767583344" Oct 30 05:09:47.174368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.174328 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" path="/var/lib/kubelet/pods/9c62b671-8729-4713-b94c-1c02da5e1388/volumes" Oct 30 05:09:47.175061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.175044 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd5a154-07d7-46f3-a114-6c8904afbf51" path="/var/lib/kubelet/pods/9dd5a154-07d7-46f3-a114-6c8904afbf51/volumes" Oct 30 05:09:47.219909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.219873 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:47.225996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.225965 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:47.741230 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.741200 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:47.742140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:47.742119 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:09:49.193069 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:49.193035 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:09:50.364210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:50.364173 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:09:52.632018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.631988 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:09:52.633690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.633667 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:52.633834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.633819 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c62b671-8729-4713-b94c-1c02da5e1388" containerName="istio-proxy" Oct 30 05:09:52.640391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.640372 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.650411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.650388 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:09:52.661263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.659962 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-47af37\"" Oct 30 05:09:52.694147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.694113 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:09:52.698992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.698970 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.715441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.715400 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:09:52.717474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.717447 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-47af37-dockercfg-qs5zr\"" Oct 30 05:09:52.793364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.777288 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gp8\" (UniqueName: \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.799085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.799045 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.799367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.799341 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.804227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.799656 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7shm\" (UniqueName: \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.956242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.927615 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gp8\" (UniqueName: \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.958726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.958698 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.975384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.975344 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.975754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.975730 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7shm\" (UniqueName: \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.976384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.959873 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:52.988240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.977651 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:52.988547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:52.986822 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gp8\" (UniqueName: \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\") pod \"payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:53.021559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:53.021523 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:53.060296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:53.043017 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7shm\" (UniqueName: \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\") pod \"payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:53.266434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:53.266403 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:53.468693 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:53.468665 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bfa743_8e65_47b2_90ba_bf7c4eb99f59.slice/crio-d52272b2760c1a5989469581d094451ca5c3fbedbc45d2dffe4c2803a580ca55 WatchSource:0}: Error finding container d52272b2760c1a5989469581d094451ca5c3fbedbc45d2dffe4c2803a580ca55: Status 404 returned error can't find the container with id d52272b2760c1a5989469581d094451ca5c3fbedbc45d2dffe4c2803a580ca55 Oct 30 05:09:53.470077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:53.470052 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:09:53.579880 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:09:53.579852 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e51776_a819_4fe1_891e_522f0dd8b946.slice/crio-0cfeeeabc56392d6cff4890a376b1bb9391e37e116a65522bcfba06df4e2520f WatchSource:0}: Error finding container 0cfeeeabc56392d6cff4890a376b1bb9391e37e116a65522bcfba06df4e2520f: Status 404 returned error can't find the container with id 0cfeeeabc56392d6cff4890a376b1bb9391e37e116a65522bcfba06df4e2520f Oct 30 05:09:53.580036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:53.580013 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:09:54.000710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.000671 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" event={"ID":"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59","Type":"ContainerStarted","Data":"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a"} Oct 30 05:09:54.001200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.000736 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" event={"ID":"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59","Type":"ContainerStarted","Data":"d52272b2760c1a5989469581d094451ca5c3fbedbc45d2dffe4c2803a580ca55"} Oct 30 05:09:54.001200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.000764 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:09:54.002772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.002749 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" event={"ID":"29e51776-a819-4fe1-891e-522f0dd8b946","Type":"ContainerStarted","Data":"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee"} Oct 30 05:09:54.002897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.002788 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" event={"ID":"29e51776-a819-4fe1-891e-522f0dd8b946","Type":"ContainerStarted","Data":"0cfeeeabc56392d6cff4890a376b1bb9391e37e116a65522bcfba06df4e2520f"} Oct 30 05:09:54.002897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.002810 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:09:54.047085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.047023 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" podStartSLOduration=2.047006347 podStartE2EDuration="2.047006347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:54.044857093 +0000 UTC m=+1567.014352095" watchObservedRunningTime="2026-10-30 05:09:54.047006347 +0000 UTC m=+1567.016501336" Oct 30 05:09:54.086442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:54.086391 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" podStartSLOduration=2.086376552 podStartE2EDuration="2.086376552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:54.085311868 +0000 UTC m=+1567.054806871" watchObservedRunningTime="2026-10-30 05:09:54.086376552 +0000 UTC m=+1567.055871539" Oct 30 05:09:55.365017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.364971 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" podUID="629bb872-a2d2-4017-9e32-8ae212d0396a" containerName="payload-processing" containerID="cri-o://81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3" gracePeriod=25 Oct 30 05:09:55.666470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.666435 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:09:55.760203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.760168 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\") pod \"629bb872-a2d2-4017-9e32-8ae212d0396a\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " Oct 30 05:09:55.760414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.760213 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\" (UniqueName: \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\") pod \"629bb872-a2d2-4017-9e32-8ae212d0396a\" (UID: \"629bb872-a2d2-4017-9e32-8ae212d0396a\") " Oct 30 05:09:55.760582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.760554 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume" pod "629bb872-a2d2-4017-9e32-8ae212d0396a" (UID: "629bb872-a2d2-4017-9e32-8ae212d0396a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:55.761137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.761116 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bb872-a2d2-4017-9e32-8ae212d0396a-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:55.762708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.762637 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt" pod "629bb872-a2d2-4017-9e32-8ae212d0396a" (UID: "629bb872-a2d2-4017-9e32-8ae212d0396a"). InnerVolumeSpecName "kube-api-access-9kqbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:55.863050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:55.863015 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqbt\" (UniqueName: \"kubernetes.io/projected/629bb872-a2d2-4017-9e32-8ae212d0396a-kube-api-access-9kqbt\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:09:56.020299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.020265 5058 generic.go:356] "Generic (PLEG): container finished" podID="629bb872-a2d2-4017-9e32-8ae212d0396a" containerID="81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3" exitCode=0 Oct 30 05:09:56.020483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.020314 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" Oct 30 05:09:56.020483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.020346 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" event={"ID":"629bb872-a2d2-4017-9e32-8ae212d0396a","Type":"ContainerDied","Data":"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3"} Oct 30 05:09:56.020483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.020397 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v" event={"ID":"629bb872-a2d2-4017-9e32-8ae212d0396a","Type":"ContainerDied","Data":"bff72190ea1b5cde052f6a73f20cd328cf1892a02511197cb117fa89e3115422"} Oct 30 05:09:56.020483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.020419 5058 scope.go:122] "RemoveContainer" containerID="81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3" Oct 30 05:09:56.034349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.034325 5058 scope.go:122] "RemoveContainer" containerID="81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3" Oct 30 05:09:56.034677 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:09:56.034657 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3\": container with ID starting with 81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3 not found: ID does not exist" containerID="81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3" Oct 30 05:09:56.034758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.034690 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3"} err="failed to get container status \"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3\": rpc error: code = NotFound desc = could not find container \"81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3\": container with ID starting with 81467f10b3a9639cbbe6edca67edd392aec75bfdf6aaad14dfbc2b467f85a7a3 not found: ID does not exist" Oct 30 05:09:56.076966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.076931 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:09:56.090186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:56.090154 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v"] Oct 30 05:09:57.215242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:09:57.214686 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629bb872-a2d2-4017-9e32-8ae212d0396a" path="/var/lib/kubelet/pods/629bb872-a2d2-4017-9e32-8ae212d0396a/volumes" Oct 30 05:10:00.688403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.688323 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:10:00.689544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.689524 5058 cpu_manager.go:414] "containerMap: removing container" podUID="629bb872-a2d2-4017-9e32-8ae212d0396a" containerName="payload-processing" Oct 30 05:10:00.689621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.689547 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629bb872-a2d2-4017-9e32-8ae212d0396a" containerName="payload-processing" Oct 30 05:10:00.689799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.689787 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="629bb872-a2d2-4017-9e32-8ae212d0396a" containerName="payload-processing" Oct 30 05:10:00.695918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.695893 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698530 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698623 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698753 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698725 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698759 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698817 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhp7\" (UniqueName: \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.698942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.698880 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.711253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.710587 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:10:00.716465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.716438 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ac96f3\"/\"kube-root-ca.crt\"" Oct 30 05:10:00.716594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.716475 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ac96f3\"/\"openshift-service-ca.crt\"" Oct 30 05:10:00.727481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.727443 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ac96f3\"/\"test-model-ac96f3-kserve-self-signed-certs\"" Oct 30 05:10:00.727617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.727504 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ac96f3\"/\"default-dockercfg-l9qrv\"" Oct 30 05:10:00.800085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800053 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800134 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800273 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800306 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800369 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhp7\" (UniqueName: \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800465 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800573 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800532 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.800687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.800669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.802718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.802691 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.802867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.802850 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:00.822572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:00.822541 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhp7\" (UniqueName: \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\") pod \"test-model-ac96f3-kserve-8476d84447-hlltd\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:01.029299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:01.022548 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:01.414346 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:01.414312 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01f3acba_c988_41b2_bbd5_6fc61e1d8323.slice/crio-2c29fe6b065081907890b7c866a623617cfd19782e80a6f650ed79d918932b24 WatchSource:0}: Error finding container 2c29fe6b065081907890b7c866a623617cfd19782e80a6f650ed79d918932b24: Status 404 returned error can't find the container with id 2c29fe6b065081907890b7c866a623617cfd19782e80a6f650ed79d918932b24 Oct 30 05:10:01.416722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:01.416696 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:10:02.111719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:02.111681 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" event={"ID":"01f3acba-c988-41b2-bbd5-6fc61e1d8323","Type":"ContainerStarted","Data":"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958"} Oct 30 05:10:02.112105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:02.111734 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" event={"ID":"01f3acba-c988-41b2-bbd5-6fc61e1d8323","Type":"ContainerStarted","Data":"2c29fe6b065081907890b7c866a623617cfd19782e80a6f650ed79d918932b24"} Oct 30 05:10:02.157997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:02.157945 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podStartSLOduration=2.157930498 podStartE2EDuration="2.157930498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:02.155335343 +0000 UTC m=+1575.124830330" watchObservedRunningTime="2026-10-30 05:10:02.157930498 +0000 UTC m=+1575.127425486" Oct 30 05:10:03.023417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:03.023389 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:04.732471 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.732437 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-02f73b"] Oct 30 05:10:04.737717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.737695 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:04.750051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.750024 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-02f73b"] Oct 30 05:10:04.757860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.757833 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\"" Oct 30 05:10:04.779592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.779558 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gnq\" (UniqueName: \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\") pod \"test-curl-8876-02f73b\" (UID: \"772206bc-c78b-4e89-bde9-8ed1b3a74150\") " pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:04.881484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.881444 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gnq\" (UniqueName: \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\") pod \"test-curl-8876-02f73b\" (UID: \"772206bc-c78b-4e89-bde9-8ed1b3a74150\") " pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:04.901659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:04.901620 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gnq\" (UniqueName: \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\") pod \"test-curl-8876-02f73b\" (UID: \"772206bc-c78b-4e89-bde9-8ed1b3a74150\") " pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:05.014994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:05.014924 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:10:05.014994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:05.014988 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:10:05.054527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:05.054491 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:05.370898 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:05.370867 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772206bc_c78b_4e89_bde9_8ed1b3a74150.slice/crio-55208c98f76a71676feda8c550107785aa9765d60f54e6563930fe081fd195cd WatchSource:0}: Error finding container 55208c98f76a71676feda8c550107785aa9765d60f54e6563930fe081fd195cd: Status 404 returned error can't find the container with id 55208c98f76a71676feda8c550107785aa9765d60f54e6563930fe081fd195cd Oct 30 05:10:05.374260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:05.374210 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-02f73b"] Oct 30 05:10:06.184278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:06.184241 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-02f73b" event={"ID":"772206bc-c78b-4e89-bde9-8ed1b3a74150","Type":"ContainerStarted","Data":"fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a"} Oct 30 05:10:06.184809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:06.184787 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-02f73b" event={"ID":"772206bc-c78b-4e89-bde9-8ed1b3a74150","Type":"ContainerStarted","Data":"55208c98f76a71676feda8c550107785aa9765d60f54e6563930fe081fd195cd"} Oct 30 05:10:06.245244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:06.243295 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-02f73b" podStartSLOduration=2.243272802 podStartE2EDuration="2.243272802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:06.241426718 +0000 UTC m=+1579.210921706" watchObservedRunningTime="2026-10-30 05:10:06.243272802 +0000 UTC m=+1579.212767796" Oct 30 05:10:07.206171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:07.206140 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-02f73b"] Oct 30 05:10:07.211499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:07.211464 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-02f73b"] Oct 30 05:10:08.213244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:08.209190 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-02f73b" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" containerName="test-curl-8876-02f73b" containerID="cri-o://fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a" gracePeriod=2 Oct 30 05:10:08.224517 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:08.224474 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-02f73b\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:09.499772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.499732 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 05:10:09.500317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.499994 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" containerID="cri-o://26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c" gracePeriod=30 Oct 30 05:10:09.513879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.513840 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 05:10:09.514159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.514127 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" containerID="cri-o://1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0" gracePeriod=30 Oct 30 05:10:09.519868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.519836 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:10:09.520121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.520076 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" containerID="cri-o://55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d" gracePeriod=30 Oct 30 05:10:09.527116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.527089 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:10:09.527404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:09.527356 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" containerID="cri-o://13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f" gracePeriod=30 Oct 30 05:10:10.440658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:10.440631 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:10.501810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:10.501726 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\" (UniqueName: \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\") pod \"772206bc-c78b-4e89-bde9-8ed1b3a74150\" (UID: \"772206bc-c78b-4e89-bde9-8ed1b3a74150\") " Oct 30 05:10:10.503890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:10.503862 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq" pod "772206bc-c78b-4e89-bde9-8ed1b3a74150" (UID: "772206bc-c78b-4e89-bde9-8ed1b3a74150"). InnerVolumeSpecName "kube-api-access-98gnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:10.603845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:10.603799 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98gnq\" (UniqueName: \"kubernetes.io/projected/772206bc-c78b-4e89-bde9-8ed1b3a74150-kube-api-access-98gnq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:11.024073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.024030 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:11.036846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.036821 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:11.123126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.123088 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:10:11.123978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.123955 5058 cpu_manager.go:414] "containerMap: removing container" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" containerName="test-curl-8876-02f73b" Oct 30 05:10:11.123978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.123978 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" containerName="test-curl-8876-02f73b" Oct 30 05:10:11.124253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.124205 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" containerName="test-curl-8876-02f73b" Oct 30 05:10:11.128279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.128257 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.146802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.146770 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:10:11.161628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.161576 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3d55bd-serving-cert\"" Oct 30 05:10:11.176535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.176501 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772206bc-c78b-4e89-bde9-8ed1b3a74150" path="/var/lib/kubelet/pods/772206bc-c78b-4e89-bde9-8ed1b3a74150/volumes" Oct 30 05:10:11.212834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.212804 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.213212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.213181 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.213433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.213398 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqncc\" (UniqueName: \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.213546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.213526 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.252983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.252946 5058 generic.go:356] "Generic (PLEG): container finished" podID="772206bc-c78b-4e89-bde9-8ed1b3a74150" containerID="fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a" exitCode=137 Oct 30 05:10:11.253174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.252993 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-02f73b" Oct 30 05:10:11.253174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.253050 5058 scope.go:122] "RemoveContainer" containerID="fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a" Oct 30 05:10:11.264878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.264857 5058 scope.go:122] "RemoveContainer" containerID="fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a" Oct 30 05:10:11.265164 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:11.265143 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a\": container with ID starting with fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a not found: ID does not exist" containerID="fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a" Oct 30 05:10:11.265239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.265175 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a"} err="failed to get container status \"fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a\": rpc error: code = NotFound desc = could not find container \"fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a\": container with ID starting with fc1d464f00940f4d084b509f82a2f1ba8fb978df763468c94fe072a5a3c9af4a not found: ID does not exist" Oct 30 05:10:11.266434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.266414 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:10:11.314661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.314574 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqncc\" (UniqueName: \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.314661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.314638 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.314854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.314719 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.314854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.314804 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.315679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.315530 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.317347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.317318 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.317521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.317505 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.334956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.334929 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqncc\" (UniqueName: \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\") pod \"maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.447241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.444552 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:11.874853 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:11.874816 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c6636_48fe_4a83_b0c8_344e43e2b069.slice/crio-8e3d23ee3c852fc34f798a39d2415e6c548170b1d589ad87a33c2073664a9a59 WatchSource:0}: Error finding container 8e3d23ee3c852fc34f798a39d2415e6c548170b1d589ad87a33c2073664a9a59: Status 404 returned error can't find the container with id 8e3d23ee3c852fc34f798a39d2415e6c548170b1d589ad87a33c2073664a9a59 Oct 30 05:10:11.876791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.876772 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:10:11.877281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:11.877258 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:10:12.263167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:12.263135 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" event={"ID":"654c6636-48fe-4a83-b0c8-344e43e2b069","Type":"ContainerStarted","Data":"8e3d23ee3c852fc34f798a39d2415e6c548170b1d589ad87a33c2073664a9a59"} Oct 30 05:10:12.882452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:12.882388 5058 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:10:12.882944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:12.882476 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:10:13.275101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:13.275058 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" event={"ID":"654c6636-48fe-4a83-b0c8-344e43e2b069","Type":"ContainerStarted","Data":"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22"} Oct 30 05:10:13.275320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:13.275126 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:13.319893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:13.319832 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" podStartSLOduration=1.955590175 podStartE2EDuration="2.319817358s" totalImagesPullingTime="364.227183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:13.316188088 +0000 UTC m=+1586.285683075" watchObservedRunningTime="2026-10-30 05:10:13.319817358 +0000 UTC m=+1586.289312340" Oct 30 05:10:14.840091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.840058 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 05:10:14.874003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.873961 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.947815 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.947886 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.947922 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.947967 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.948016 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.948047 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.948178 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.948272 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.948385 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\") pod \"86b40eef-545d-4cc5-8ca3-3de7ff375188\" (UID: \"86b40eef-545d-4cc5-8ca3-3de7ff375188\") " Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.951673 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.951964 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.952020 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:14.952242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.952066 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:14.953745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.952355 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:14.955291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.955263 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:14.956628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.956601 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:14.956762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.956736 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "kube-api-access-dpdxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:14.957793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.957380 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token" pod "86b40eef-545d-4cc5-8ca3-3de7ff375188" (UID: "86b40eef-545d-4cc5-8ca3-3de7ff375188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:14.976774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.976736 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:10:14.980621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:14.980592 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 05:10:15.050540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050507 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050561 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050599 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050642 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050747 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050793 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050845 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.050907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.050878 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.051118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051070 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\") pod \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\" (UID: \"17d00444-0a9b-464c-a0d3-d2f0b300c7b3\") " Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051208 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051506 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051757 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051942 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051954 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.051963 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpdxh\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-kube-api-access-dpdxh\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052059 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052115 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052143 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052158 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052186 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052202 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052213 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052253 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b40eef-545d-4cc5-8ca3-3de7ff375188-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052265 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b40eef-545d-4cc5-8ca3-3de7ff375188-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052277 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.052336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.052289 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b40eef-545d-4cc5-8ca3-3de7ff375188-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.053805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.053778 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.053897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.053834 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "kube-api-access-t84r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.054259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.054201 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.055434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.055409 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo" pod "17d00444-0a9b-464c-a0d3-d2f0b300c7b3" (UID: "17d00444-0a9b-464c-a0d3-d2f0b300c7b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:15.153131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153098 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.153131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153141 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153184 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153230 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153288 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153328 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153379 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153418 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.153528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153475 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153544 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153583 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153608 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153719 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153751 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153778 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153864 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153901 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153921 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.154040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.153961 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\") pod \"9dc22070-efee-4345-8efe-ab9646336281\" (UID: \"9dc22070-efee-4345-8efe-ab9646336281\") " Oct 30 05:10:15.154557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154049 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\") pod \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\" (UID: \"c2dfe251-99ad-4b6f-9b5b-3747c3b334be\") " Oct 30 05:10:15.154557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154101 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.154557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154421 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:15.154763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154743 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84r6\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-kube-api-access-t84r6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154768 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154775 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154784 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154818 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154832 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154845 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d00444-0a9b-464c-a0d3-d2f0b300c7b3-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154856 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.154866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154867 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154879 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154891 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc22070-efee-4345-8efe-ab9646336281-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.154959 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.156051 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.156121 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.156184 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.156920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.155124 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.157367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.157054 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.157367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.157318 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.159336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.159310 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:15.159336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.159322 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "kube-api-access-fsxkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.159503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.159397 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy" pod "c2dfe251-99ad-4b6f-9b5b-3747c3b334be" (UID: "c2dfe251-99ad-4b6f-9b5b-3747c3b334be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.159503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.159402 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "kube-api-access-zklv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.160980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.160959 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.161110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.161083 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo" pod "9dc22070-efee-4345-8efe-ab9646336281" (UID: "9dc22070-efee-4345-8efe-ab9646336281"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259461 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259503 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259517 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc22070-efee-4345-8efe-ab9646336281-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259530 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259543 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259555 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259567 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259580 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259592 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259604 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259616 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsxkq\" (UniqueName: \"kubernetes.io/projected/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-kube-api-access-fsxkq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259630 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc22070-efee-4345-8efe-ab9646336281-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259644 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklv9\" (UniqueName: \"kubernetes.io/projected/9dc22070-efee-4345-8efe-ab9646336281-kube-api-access-zklv9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.271699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.259655 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dfe251-99ad-4b6f-9b5b-3747c3b334be-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.291890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.291858 5058 generic.go:356] "Generic (PLEG): container finished" podID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerID="55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d" exitCode=0 Oct 30 05:10:15.292074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.291906 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" Oct 30 05:10:15.292074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.291935 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" event={"ID":"17d00444-0a9b-464c-a0d3-d2f0b300c7b3","Type":"ContainerDied","Data":"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d"} Oct 30 05:10:15.292074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.291990 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c" event={"ID":"17d00444-0a9b-464c-a0d3-d2f0b300c7b3","Type":"ContainerDied","Data":"b4bfa5fdba4650631f7b988cfede0ab8bce94bedcf480e2111a91a34290ddb9f"} Oct 30 05:10:15.292074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.292012 5058 scope.go:122] "RemoveContainer" containerID="55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d" Oct 30 05:10:15.293598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.293573 5058 generic.go:356] "Generic (PLEG): container finished" podID="9dc22070-efee-4345-8efe-ab9646336281" containerID="13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f" exitCode=0 Oct 30 05:10:15.293720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.293635 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" Oct 30 05:10:15.293720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.293659 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" event={"ID":"9dc22070-efee-4345-8efe-ab9646336281","Type":"ContainerDied","Data":"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f"} Oct 30 05:10:15.293720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.293708 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676" event={"ID":"9dc22070-efee-4345-8efe-ab9646336281","Type":"ContainerDied","Data":"20f4babe7801a42ce1191069fa87a019c1c215a24e492aa84de4a2e9457d8acb"} Oct 30 05:10:15.296429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.296402 5058 generic.go:356] "Generic (PLEG): container finished" podID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerID="1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0" exitCode=0 Oct 30 05:10:15.296611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.296594 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerDied","Data":"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0"} Oct 30 05:10:15.296690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.296637 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" event={"ID":"86b40eef-545d-4cc5-8ca3-3de7ff375188","Type":"ContainerDied","Data":"dbe8c28a40b20f13e593162f4a854b3c8975f3dd26926806587b5ff1d6ddb875"} Oct 30 05:10:15.296756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.296705 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd" Oct 30 05:10:15.298588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.298568 5058 generic.go:356] "Generic (PLEG): container finished" podID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerID="26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c" exitCode=0 Oct 30 05:10:15.298679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.298593 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" event={"ID":"c2dfe251-99ad-4b6f-9b5b-3747c3b334be","Type":"ContainerDied","Data":"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c"} Oct 30 05:10:15.298679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.298631 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" event={"ID":"c2dfe251-99ad-4b6f-9b5b-3747c3b334be","Type":"ContainerDied","Data":"294d00a94f431c5d2263522f93f0fc9adc94fd700287cb2e5ea66d15b2df0ffb"} Oct 30 05:10:15.298760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.298679 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58" Oct 30 05:10:15.308391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.308363 5058 scope.go:122] "RemoveContainer" containerID="55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d" Oct 30 05:10:15.309095 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:15.309069 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d\": container with ID starting with 55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d not found: ID does not exist" containerID="55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d" Oct 30 05:10:15.309181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.309101 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d"} err="failed to get container status \"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d\": rpc error: code = NotFound desc = could not find container \"55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d\": container with ID starting with 55360bea2965609c6077171a06efea8c3992fee80674cc0c55c123c1258c682d not found: ID does not exist" Oct 30 05:10:15.309181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.309122 5058 scope.go:122] "RemoveContainer" containerID="13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f" Oct 30 05:10:15.320800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.320782 5058 scope.go:122] "RemoveContainer" containerID="13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f" Oct 30 05:10:15.321048 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:15.321031 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f\": container with ID starting with 13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f not found: ID does not exist" containerID="13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f" Oct 30 05:10:15.321085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.321057 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f"} err="failed to get container status \"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f\": rpc error: code = NotFound desc = could not find container \"13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f\": container with ID starting with 13677b58dccc0cc27ca5d8f36d5df448f5b52e9912310acd8b8a89856881cd5f not found: ID does not exist" Oct 30 05:10:15.321085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.321073 5058 scope.go:122] "RemoveContainer" containerID="1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0" Oct 30 05:10:15.331464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.331439 5058 scope.go:122] "RemoveContainer" containerID="4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18" Oct 30 05:10:15.341708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.341688 5058 scope.go:122] "RemoveContainer" containerID="1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0" Oct 30 05:10:15.341983 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:15.341963 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0\": container with ID starting with 1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0 not found: ID does not exist" containerID="1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0" Oct 30 05:10:15.342045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.341991 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0"} err="failed to get container status \"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0\": rpc error: code = NotFound desc = could not find container \"1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0\": container with ID starting with 1caded592fd95a8dd582013658959be0638fec0048e909460d14898d8eab40a0 not found: ID does not exist" Oct 30 05:10:15.342045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.342011 5058 scope.go:122] "RemoveContainer" containerID="4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18" Oct 30 05:10:15.342255 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:15.342240 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18\": container with ID starting with 4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18 not found: ID does not exist" containerID="4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18" Oct 30 05:10:15.342301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.342258 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18"} err="failed to get container status \"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18\": rpc error: code = NotFound desc = could not find container \"4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18\": container with ID starting with 4de944b922be84c3f881916338fed9ed0b8b2cb40a3332687f43b7a8b00a4d18 not found: ID does not exist" Oct 30 05:10:15.342301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.342269 5058 scope.go:122] "RemoveContainer" containerID="26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c" Oct 30 05:10:15.353150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.353121 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 05:10:15.353729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.353709 5058 scope.go:122] "RemoveContainer" containerID="26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c" Oct 30 05:10:15.354069 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:15.354043 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c\": container with ID starting with 26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c not found: ID does not exist" containerID="26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c" Oct 30 05:10:15.354147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.354080 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c"} err="failed to get container status \"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c\": rpc error: code = NotFound desc = could not find container \"26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c\": container with ID starting with 26a687ef2bdac2d2359f82d4d4e65ff0bd89db8f0c5e2d92f298f455b2d9f80c not found: ID does not exist" Oct 30 05:10:15.366024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.365997 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd"] Oct 30 05:10:15.403030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.402995 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:10:15.406678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.406626 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676"] Oct 30 05:10:15.442389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.442354 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:10:15.452414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.452384 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c"] Oct 30 05:10:15.488213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.488182 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 05:10:15.492737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:15.492700 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58"] Oct 30 05:10:17.198242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.194617 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" path="/var/lib/kubelet/pods/17d00444-0a9b-464c-a0d3-d2f0b300c7b3/volumes" Oct 30 05:10:17.198242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.195414 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" path="/var/lib/kubelet/pods/86b40eef-545d-4cc5-8ca3-3de7ff375188/volumes" Oct 30 05:10:17.223248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.220982 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc22070-efee-4345-8efe-ab9646336281" path="/var/lib/kubelet/pods/9dc22070-efee-4345-8efe-ab9646336281/volumes" Oct 30 05:10:17.223248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.222173 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" path="/var/lib/kubelet/pods/c2dfe251-99ad-4b6f-9b5b-3747c3b334be/volumes" Oct 30 05:10:17.907883 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.907851 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:10:17.941293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:17.941260 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:10:18.143028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.142989 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:10:18.144144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144116 5058 cpu_manager.go:414] "containerMap: removing container" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:18.144144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144139 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144153 5058 cpu_manager.go:414] "containerMap: removing container" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144163 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144252 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144260 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144275 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" Oct 30 05:10:18.144377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144283 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" Oct 30 05:10:18.144723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144436 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:18.144723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144454 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2dfe251-99ad-4b6f-9b5b-3747c3b334be" containerName="istio-proxy" Oct 30 05:10:18.144723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144465 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc22070-efee-4345-8efe-ab9646336281" containerName="istio-proxy" Oct 30 05:10:18.144723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.144479 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d00444-0a9b-464c-a0d3-d2f0b300c7b3" containerName="istio-proxy" Oct 30 05:10:18.151291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.151249 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.168759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.168730 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0814c-openshift-default-dockercfg-6qdsn\"" Oct 30 05:10:18.174196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.174171 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:10:18.257182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257141 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.257633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257290 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.257633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257332 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.257633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257464 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.257633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257618 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skx5\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.258705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257746 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.258705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257844 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.258705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257912 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.258705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.257938 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359451 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359525 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359544 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359584 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359629 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359662 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359731 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359821 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skx5\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.359886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359867 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.360096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.359955 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.360096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.360047 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.360206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.360172 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.360270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.360241 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.360479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.360456 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.362164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.362142 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.362480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.362463 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.380386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.380359 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.380771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.380750 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skx5\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\") pod \"e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.477839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.477755 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:18.996843 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:18.996797 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dda135_a34a_4e91_8175_5bd3c53ae67b.slice/crio-22f8cac94d75d1229db6cb8894c8d4796695bdcb057739b516fb7c1b34f79bae WatchSource:0}: Error finding container 22f8cac94d75d1229db6cb8894c8d4796695bdcb057739b516fb7c1b34f79bae: Status 404 returned error can't find the container with id 22f8cac94d75d1229db6cb8894c8d4796695bdcb057739b516fb7c1b34f79bae Oct 30 05:10:18.998276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.998210 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:10:18.999785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.999753 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:18.999884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.999819 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:18.999884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:18.999847 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:19.193004 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:19.192971 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:10:19.303786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.303574 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:10:19.345848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.345799 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerStarted","Data":"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db"} Oct 30 05:10:19.346092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.345868 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerStarted","Data":"22f8cac94d75d1229db6cb8894c8d4796695bdcb057739b516fb7c1b34f79bae"} Oct 30 05:10:19.395260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.395165 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podStartSLOduration=1.395143454 podStartE2EDuration="1.395143454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:19.389689774 +0000 UTC m=+1592.359184786" watchObservedRunningTime="2026-10-30 05:10:19.395143454 +0000 UTC m=+1592.364638442" Oct 30 05:10:19.478517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.478481 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:19.995444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.995412 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:10:19.996258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.996239 5058 cpu_manager.go:414] "containerMap: removing container" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:19.996258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.996256 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:19.996435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:19.996396 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b40eef-545d-4cc5-8ca3-3de7ff375188" containerName="istio-proxy" Oct 30 05:10:20.011921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.011179 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.019464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.019436 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:10:20.029697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.029669 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c613f5ea-openshift-default-dockercfg-dxwcw\"" Oct 30 05:10:20.030522 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.030479 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:10:20.030778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.030604 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.035911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.035888 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:10:20.108921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.108881 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.108921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.108928 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.108985 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109019 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109038 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvlb\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109136 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109188 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kck5z\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109207 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109285 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109348 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109387 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109519 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109540 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109566 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109599 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109649 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.109714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109703 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.109761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.109722 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.211543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211511 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck5z\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.211543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211550 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.211733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.211733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211705 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.211855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211767 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.211914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211903 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.211966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211923 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.211966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211958 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.211981 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212000 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212036 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212135 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212161 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212236 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212273 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212301 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212312 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212347 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212399 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212430 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvlb\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212476 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212573 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212655 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212677 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212662 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212706 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.212990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212828 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.212990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.212977 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.214330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.214301 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.214625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.214604 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.214900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.214879 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.215090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.215072 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.232295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.232254 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.232484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.232463 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck5z\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.232785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.232763 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvlb\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.233861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.233844 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.338804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.338713 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:20.350278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.350011 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:20.485083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.485053 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:20.681777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.681744 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:10:20.699963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.699924 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:20.700089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.700013 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:20.700089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.700059 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:20.701125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.701106 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:10:20.702306 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:20.702260 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f1fd6f_0bee_4ad6_8b36_579d6eaecb46.slice/crio-171e6abe6bc1482c3d7283d00e4e16c74b41035aaad596fa06ee73ab840ce126 WatchSource:0}: Error finding container 171e6abe6bc1482c3d7283d00e4e16c74b41035aaad596fa06ee73ab840ce126: Status 404 returned error can't find the container with id 171e6abe6bc1482c3d7283d00e4e16c74b41035aaad596fa06ee73ab840ce126 Oct 30 05:10:20.711560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.711521 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:20.711645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.711608 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:20.711691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:20.711650 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:21.378970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.378932 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" event={"ID":"d715f702-dbb5-4595-a978-152131f5da48","Type":"ContainerStarted","Data":"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de"} Oct 30 05:10:21.379494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.378996 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" event={"ID":"d715f702-dbb5-4595-a978-152131f5da48","Type":"ContainerStarted","Data":"62608cb300e9549f1c2082bb92ba691a37f3cc2c7551277980f8edd87dbd0f2b"} Oct 30 05:10:21.380848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.380823 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" event={"ID":"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46","Type":"ContainerStarted","Data":"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5"} Oct 30 05:10:21.381006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.380872 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" event={"ID":"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46","Type":"ContainerStarted","Data":"171e6abe6bc1482c3d7283d00e4e16c74b41035aaad596fa06ee73ab840ce126"} Oct 30 05:10:21.381256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.381203 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:21.382169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.382149 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:10:21.429297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.429244 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podStartSLOduration=2.429229035 podStartE2EDuration="2.429229035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:21.427279401 +0000 UTC m=+1594.396774388" watchObservedRunningTime="2026-10-30 05:10:21.429229035 +0000 UTC m=+1594.398724016" Oct 30 05:10:21.522459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:21.522408 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podStartSLOduration=2.522393205 podStartE2EDuration="2.522393205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:21.517437932 +0000 UTC m=+1594.486932933" watchObservedRunningTime="2026-10-30 05:10:21.522393205 +0000 UTC m=+1594.491888207" Oct 30 05:10:22.347885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.347853 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:22.348275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.348249 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:22.348452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.348427 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:22.387261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.387181 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:22.395573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.388608 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:22.395573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.395404 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:22.908454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.908411 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" podUID="728fc409-6170-4b53-9671-8c98543dc33d" containerName="payload-processing" containerID="cri-o://13dae93687929b71d4bd51f1201f0e8045c6d816c75575f19fea7ea9693744eb" gracePeriod=25 Oct 30 05:10:22.961289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:22.942906 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" podUID="11a7da68-6287-46e3-a088-9ff9675c9276" containerName="payload-pre-processing" containerID="cri-o://2bdfb5910a6165664509993c0eec27bfb8686b22385ad17f0114844d084abc72" gracePeriod=25 Oct 30 05:10:23.397254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.389977 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:23.512353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.512255 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:23.514200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.425546 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:23.514417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.514392 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:23.522360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.522333 5058 generic.go:356] "Generic (PLEG): container finished" podID="728fc409-6170-4b53-9671-8c98543dc33d" containerID="13dae93687929b71d4bd51f1201f0e8045c6d816c75575f19fea7ea9693744eb" exitCode=0 Oct 30 05:10:23.522659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.522401 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" event={"ID":"728fc409-6170-4b53-9671-8c98543dc33d","Type":"ContainerDied","Data":"13dae93687929b71d4bd51f1201f0e8045c6d816c75575f19fea7ea9693744eb"} Oct 30 05:10:23.588579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.588539 5058 generic.go:356] "Generic (PLEG): container finished" podID="11a7da68-6287-46e3-a088-9ff9675c9276" containerID="2bdfb5910a6165664509993c0eec27bfb8686b22385ad17f0114844d084abc72" exitCode=0 Oct 30 05:10:23.589863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.589834 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" event={"ID":"11a7da68-6287-46e3-a088-9ff9675c9276","Type":"ContainerDied","Data":"2bdfb5910a6165664509993c0eec27bfb8686b22385ad17f0114844d084abc72"} Oct 30 05:10:23.600274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.599491 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:10:23.784731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.783865 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume" pod "728fc409-6170-4b53-9671-8c98543dc33d" (UID: "728fc409-6170-4b53-9671-8c98543dc33d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:23.784731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.783445 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\") pod \"728fc409-6170-4b53-9671-8c98543dc33d\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " Oct 30 05:10:23.784731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.784006 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\" (UniqueName: \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\") pod \"728fc409-6170-4b53-9671-8c98543dc33d\" (UID: \"728fc409-6170-4b53-9671-8c98543dc33d\") " Oct 30 05:10:23.785276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.785132 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728fc409-6170-4b53-9671-8c98543dc33d-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.787546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.787513 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl" pod "728fc409-6170-4b53-9671-8c98543dc33d" (UID: "728fc409-6170-4b53-9671-8c98543dc33d"). InnerVolumeSpecName "kube-api-access-vzztl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:23.794175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.794154 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:10:23.889242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.887808 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\") pod \"11a7da68-6287-46e3-a088-9ff9675c9276\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " Oct 30 05:10:23.889242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.888145 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\" (UniqueName: \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\") pod \"11a7da68-6287-46e3-a088-9ff9675c9276\" (UID: \"11a7da68-6287-46e3-a088-9ff9675c9276\") " Oct 30 05:10:23.889242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.888325 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume" pod "11a7da68-6287-46e3-a088-9ff9675c9276" (UID: "11a7da68-6287-46e3-a088-9ff9675c9276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:23.889242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.888820 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzztl\" (UniqueName: \"kubernetes.io/projected/728fc409-6170-4b53-9671-8c98543dc33d-kube-api-access-vzztl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.889242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.888846 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a7da68-6287-46e3-a088-9ff9675c9276-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.891151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.891123 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf" pod "11a7da68-6287-46e3-a088-9ff9675c9276" (UID: "11a7da68-6287-46e3-a088-9ff9675c9276"). InnerVolumeSpecName "kube-api-access-g8wkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:23.983107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.983070 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984569 5058 cpu_manager.go:414] "containerMap: removing container" podUID="11a7da68-6287-46e3-a088-9ff9675c9276" containerName="payload-pre-processing" Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984596 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a7da68-6287-46e3-a088-9ff9675c9276" containerName="payload-pre-processing" Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984699 5058 cpu_manager.go:414] "containerMap: removing container" podUID="728fc409-6170-4b53-9671-8c98543dc33d" containerName="payload-processing" Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984709 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728fc409-6170-4b53-9671-8c98543dc33d" containerName="payload-processing" Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984888 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a7da68-6287-46e3-a088-9ff9675c9276" containerName="payload-pre-processing" Oct 30 05:10:23.985832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.984916 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="728fc409-6170-4b53-9671-8c98543dc33d" containerName="payload-processing" Oct 30 05:10:23.989802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.989776 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:23.990000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:23.989975 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wkf\" (UniqueName: \"kubernetes.io/projected/11a7da68-6287-46e3-a088-9ff9675c9276-kube-api-access-g8wkf\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:24.008047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.007454 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b0814c\"" Oct 30 05:10:24.010320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.010081 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:10:24.123295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.091372 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2r52\" (UniqueName: \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.123295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.091474 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.123295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.110698 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:10:24.131241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.124332 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:10:24.131241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.124463 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.158276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.149123 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b0814c-dockercfg-n8lbb\"" Oct 30 05:10:24.210240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.201636 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfwd\" (UniqueName: \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.210240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.201790 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2r52\" (UniqueName: \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.210240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.201831 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.210240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.201902 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.210240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.202881 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.331365 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfwd\" (UniqueName: \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.331555 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.332835 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.333928 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2r52\" (UniqueName: \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\") pod \"payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.348645 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.367150 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.367205 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.367607 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:24.392472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.367648 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:24.541116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.530007 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfwd\" (UniqueName: \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\") pod \"payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.632242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.624213 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" Oct 30 05:10:24.661858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.661758 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh" event={"ID":"728fc409-6170-4b53-9671-8c98543dc33d","Type":"ContainerDied","Data":"f8d743d9e2c44fd30f31bd616e060d8ba6c6789fe26c5e1af2ccf1499baf54e3"} Oct 30 05:10:24.662115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.662095 5058 scope.go:122] "RemoveContainer" containerID="13dae93687929b71d4bd51f1201f0e8045c6d816c75575f19fea7ea9693744eb" Oct 30 05:10:24.695497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.695463 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:24.697418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.696584 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" event={"ID":"11a7da68-6287-46e3-a088-9ff9675c9276","Type":"ContainerDied","Data":"76a395695497856c5cfb9964d696e505cf7d9c433b753cd2b17e8ec21cea2464"} Oct 30 05:10:24.697418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.696678 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g" Oct 30 05:10:24.721855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.720735 5058 scope.go:122] "RemoveContainer" containerID="2bdfb5910a6165664509993c0eec27bfb8686b22385ad17f0114844d084abc72" Oct 30 05:10:24.802256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.802166 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:10:24.811452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.811410 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh"] Oct 30 05:10:24.821574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.821521 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:10:24.881764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:24.881723 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g"] Oct 30 05:10:25.198623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.197524 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a7da68-6287-46e3-a088-9ff9675c9276" path="/var/lib/kubelet/pods/11a7da68-6287-46e3-a088-9ff9675c9276/volumes" Oct 30 05:10:25.198623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.197931 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728fc409-6170-4b53-9671-8c98543dc33d" path="/var/lib/kubelet/pods/728fc409-6170-4b53-9671-8c98543dc33d/volumes" Oct 30 05:10:25.198623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.198308 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:10:25.198623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.198357 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:10:25.345270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.345071 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:25.345270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.345133 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:25.384033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.383918 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:25.384033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.383975 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:25.734292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.734249 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" event={"ID":"1471b201-a8bc-4b28-bd84-8bfbefbb046b","Type":"ContainerStarted","Data":"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a"} Oct 30 05:10:25.734899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.734865 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" event={"ID":"1471b201-a8bc-4b28-bd84-8bfbefbb046b","Type":"ContainerStarted","Data":"acc03e68c50cd1cd977b28ec9926a65852bda7809fbb314e8dd9603fd1bc3982"} Oct 30 05:10:25.736263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.736240 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:25.806241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.801781 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" event={"ID":"5585502a-2049-40be-98dd-ea02267c56e8","Type":"ContainerStarted","Data":"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8"} Oct 30 05:10:25.806241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.801853 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" event={"ID":"5585502a-2049-40be-98dd-ea02267c56e8","Type":"ContainerStarted","Data":"70be3568d11dc060e251510f47e35996339a75723348ebcc8c64ccc363f6ca69"} Oct 30 05:10:25.806241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.803063 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:25.828295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:25.822185 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" podStartSLOduration=1.822164904 podStartE2EDuration="1.822164904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:25.782681982 +0000 UTC m=+1598.752176983" watchObservedRunningTime="2026-10-30 05:10:25.822164904 +0000 UTC m=+1598.791659892" Oct 30 05:10:26.341210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:26.340992 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:26.341210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:26.341056 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:26.365243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:26.364831 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:26.365243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:26.364889 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:27.346241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:27.341005 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:10:27.346241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:27.341070 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:10:27.352242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:27.350935 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:10:27.352242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:27.350997 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:10:28.343791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.343757 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:28.344236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.344205 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:28.344922 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.344900 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:10:28.354822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.354796 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:28.355234 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.355176 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:28.355913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.355897 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:10:28.403685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:28.403635 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" podStartSLOduration=5.403620612 podStartE2EDuration="5.403620612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:25.851570693 +0000 UTC m=+1598.821065681" watchObservedRunningTime="2026-10-30 05:10:28.403620612 +0000 UTC m=+1601.373115600" Oct 30 05:10:30.932505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:30.932469 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:10:30.938539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:30.938515 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:30.956456 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:30.956416 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:10:30.963311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:30.963282 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-47af37-serving-cert\"" Oct 30 05:10:31.064612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.064572 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.064800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.064734 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.064900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.064878 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.065008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.064996 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jscv7\" (UniqueName: \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.173491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.173441 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.176360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.176332 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.181405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.181362 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.183610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.183528 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jscv7\" (UniqueName: \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.191054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.174659 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.191054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.185850 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.198449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.198173 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.245514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.242961 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jscv7\" (UniqueName: \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\") pod \"maas-api-e2e-api-b-47af37-565b499dff-nzspd\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.262052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.262011 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:31.580840 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:31.580811 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e82fbed_b079_4c18_aa4f_47c4f8f842ca.slice/crio-b96cbdb5709d0236445ab38b808973723a880863bc489c31816578a158965782 WatchSource:0}: Error finding container b96cbdb5709d0236445ab38b808973723a880863bc489c31816578a158965782: Status 404 returned error can't find the container with id b96cbdb5709d0236445ab38b808973723a880863bc489c31816578a158965782 Oct 30 05:10:31.582419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.582390 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:10:31.955374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:31.955316 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" event={"ID":"4e82fbed-b079-4c18-aa4f-47c4f8f842ca","Type":"ContainerStarted","Data":"b96cbdb5709d0236445ab38b808973723a880863bc489c31816578a158965782"} Oct 30 05:10:32.631861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.631821 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:10:32.638040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.638006 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.656561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.656530 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:10:32.661437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.661406 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4225df\"/\"kube-root-ca.crt\"" Oct 30 05:10:32.661437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.661426 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4225df\"/\"openshift-service-ca.crt\"" Oct 30 05:10:32.667416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.667388 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4225df\"/\"default-dockercfg-7xt6d\"" Oct 30 05:10:32.667537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.667487 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4225df\"/\"test-model-4225df-kserve-self-signed-certs\"" Oct 30 05:10:32.723957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.723920 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.723957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.723970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.724183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.724105 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.724183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.724133 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4sjw\" (UniqueName: \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.724358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.724334 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.724438 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.724425 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.825962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.825923 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.825983 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826059 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826084 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826256 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826321 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4sjw\" (UniqueName: \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826388 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826504 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.826675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.826649 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.828343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.828315 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.828657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.828627 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.890374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.884152 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4sjw\" (UniqueName: \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\") pod \"test-model-4225df-kserve-b44db4d5-kwm9v\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:32.959333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:32.958515 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:33.008254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:33.000589 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" event={"ID":"4e82fbed-b079-4c18-aa4f-47c4f8f842ca","Type":"ContainerStarted","Data":"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc"} Oct 30 05:10:33.008254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:33.001895 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:33.076243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:33.074137 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" podStartSLOduration=2.770712956 podStartE2EDuration="3.074115737s" totalImagesPullingTime="303.402781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:33.070532911 +0000 UTC m=+1606.040027900" watchObservedRunningTime="2026-10-30 05:10:33.074115737 +0000 UTC m=+1606.043610724" Oct 30 05:10:33.358493 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:33.358465 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode25df3e1_3ac5_49e8_bd47_ee15dbe7f5b2.slice/crio-61e3f7712eab6f3351ca3ae2bbb4f7790a07ec33b0e6f6267daa529208b36c2a WatchSource:0}: Error finding container 61e3f7712eab6f3351ca3ae2bbb4f7790a07ec33b0e6f6267daa529208b36c2a: Status 404 returned error can't find the container with id 61e3f7712eab6f3351ca3ae2bbb4f7790a07ec33b0e6f6267daa529208b36c2a Oct 30 05:10:33.360436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:33.360409 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:10:34.012471 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:34.012436 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" event={"ID":"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2","Type":"ContainerStarted","Data":"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1"} Oct 30 05:10:34.012958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:34.012478 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" event={"ID":"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2","Type":"ContainerStarted","Data":"61e3f7712eab6f3351ca3ae2bbb4f7790a07ec33b0e6f6267daa529208b36c2a"} Oct 30 05:10:34.055082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:34.055026 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podStartSLOduration=2.055010656 podStartE2EDuration="2.055010656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:34.053126075 +0000 UTC m=+1607.022621065" watchObservedRunningTime="2026-10-30 05:10:34.055010656 +0000 UTC m=+1607.024505644" Oct 30 05:10:34.965243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:34.960113 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:35.080159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.080121 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:10:35.086578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.086552 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.106835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.106805 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:10:35.189553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189513 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189559 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189586 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189684 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189710 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2jx\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189747 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189945 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.189996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.189980 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.190201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.190172 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291462 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291518 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291563 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291589 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2jx\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291629 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291819 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291881 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.291992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.291959 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292031 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292127 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292177 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292387 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292789 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.292898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.292794 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.294751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.294726 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.294970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.294954 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.312141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.312104 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.312298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.312202 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2jx\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.413034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.413002 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:35.738927 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:35.738894 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2295196_ca25_4d90_965a_9be42fd3b09b.slice/crio-2244ce60feac8254420e1d16a490de52fb698ed6a672301c26f09ad9b92ab99b WatchSource:0}: Error finding container 2244ce60feac8254420e1d16a490de52fb698ed6a672301c26f09ad9b92ab99b: Status 404 returned error can't find the container with id 2244ce60feac8254420e1d16a490de52fb698ed6a672301c26f09ad9b92ab99b Oct 30 05:10:35.740525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.740502 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:10:35.742120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.742089 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.742198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.742151 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.742198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:35.742181 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:36.031088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.030991 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" event={"ID":"a2295196-ca25-4d90-965a-9be42fd3b09b","Type":"ContainerStarted","Data":"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d"} Oct 30 05:10:36.031088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.031086 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" event={"ID":"a2295196-ca25-4d90-965a-9be42fd3b09b","Type":"ContainerStarted","Data":"2244ce60feac8254420e1d16a490de52fb698ed6a672301c26f09ad9b92ab99b"} Oct 30 05:10:36.081200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.081134 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podStartSLOduration=1.081117452 podStartE2EDuration="1.081117452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:36.077337303 +0000 UTC m=+1609.046832291" watchObservedRunningTime="2026-10-30 05:10:36.081117452 +0000 UTC m=+1609.050612442" Oct 30 05:10:36.413271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.413238 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:36.414706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.414682 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:36.414822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:36.414731 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:37.414337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:37.414306 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:37.414723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:37.414358 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:37.889894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:37.889862 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:10:37.890289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:37.890270 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:10:38.416240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:38.414740 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:38.416240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:38.414804 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:39.413829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:39.413795 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:39.414007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:39.413850 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:40.051326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:40.051287 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:10:40.414080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:40.414044 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:40.414312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:40.414107 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:40.697151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:40.697031 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:10:40.697450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:40.697311 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" podUID="c25f7963-5f1c-423b-a63c-6aee2121faad" containerName="maas-api" containerID="cri-o://6f23d584f01e72314a542a4d6064bf5f0c65c874120fc2eb3371af9cf8c35e78" gracePeriod=30 Oct 30 05:10:41.427123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.424678 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:41.427123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.424746 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:41.452860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.451857 5058 generic.go:356] "Generic (PLEG): container finished" podID="c25f7963-5f1c-423b-a63c-6aee2121faad" containerID="6f23d584f01e72314a542a4d6064bf5f0c65c874120fc2eb3371af9cf8c35e78" exitCode=0 Oct 30 05:10:41.452860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.452139 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" event={"ID":"c25f7963-5f1c-423b-a63c-6aee2121faad","Type":"ContainerDied","Data":"6f23d584f01e72314a542a4d6064bf5f0c65c874120fc2eb3371af9cf8c35e78"} Oct 30 05:10:41.506240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.504657 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:10:41.718837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.711729 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\") pod \"c25f7963-5f1c-423b-a63c-6aee2121faad\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " Oct 30 05:10:41.718837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.711785 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\" (UniqueName: \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\") pod \"c25f7963-5f1c-423b-a63c-6aee2121faad\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " Oct 30 05:10:41.718837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.712033 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\") pod \"c25f7963-5f1c-423b-a63c-6aee2121faad\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " Oct 30 05:10:41.718837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.712118 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\") pod \"c25f7963-5f1c-423b-a63c-6aee2121faad\" (UID: \"c25f7963-5f1c-423b-a63c-6aee2121faad\") " Oct 30 05:10:41.718837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.713176 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca" pod "c25f7963-5f1c-423b-a63c-6aee2121faad" (UID: "c25f7963-5f1c-423b-a63c-6aee2121faad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:41.763458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.751862 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs" pod "c25f7963-5f1c-423b-a63c-6aee2121faad" (UID: "c25f7963-5f1c-423b-a63c-6aee2121faad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:41.778780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.778730 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls" pod "c25f7963-5f1c-423b-a63c-6aee2121faad" (UID: "c25f7963-5f1c-423b-a63c-6aee2121faad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:41.800893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.800848 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td" pod "c25f7963-5f1c-423b-a63c-6aee2121faad" (UID: "c25f7963-5f1c-423b-a63c-6aee2121faad"). InnerVolumeSpecName "kube-api-access-7p2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:41.822461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.818135 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:41.822461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.818170 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25f7963-5f1c-423b-a63c-6aee2121faad-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:41.822461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.818184 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25f7963-5f1c-423b-a63c-6aee2121faad-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:41.822461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:41.818197 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2td\" (UniqueName: \"kubernetes.io/projected/c25f7963-5f1c-423b-a63c-6aee2121faad-kube-api-access-7p2td\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:10:42.414045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.414005 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:10:42.414386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.414352 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 05:10:42.466504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.466464 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" event={"ID":"c25f7963-5f1c-423b-a63c-6aee2121faad","Type":"ContainerDied","Data":"90125c80c04ee15259dfc8363eefa08a7179d9d681cef571f08008762ecbe988"} Oct 30 05:10:42.466890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.466535 5058 scope.go:122] "RemoveContainer" containerID="6f23d584f01e72314a542a4d6064bf5f0c65c874120fc2eb3371af9cf8c35e78" Oct 30 05:10:42.466890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.466705 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf" Oct 30 05:10:42.551398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.551348 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:10:42.558423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.558385 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c613f5ea-767554ffb7-hf6hf"] Oct 30 05:10:42.959649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.959617 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:42.973058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:42.973030 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:43.164111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:43.164074 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25f7963-5f1c-423b-a63c-6aee2121faad" path="/var/lib/kubelet/pods/c25f7963-5f1c-423b-a63c-6aee2121faad/volumes" Oct 30 05:10:43.486877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:43.486840 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:10:43.856595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:43.856496 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:43.857047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:43.857025 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:43.857908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:43.857885 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:10:48.226715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:48.226680 5058 scope.go:122] "RemoveContainer" containerID="ba70b7432309bd30269603289ee48c8a529bff4e56dd873e15a298009213dd48" Oct 30 05:10:49.192087 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:49.192046 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:10:50.120349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.120123 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:10:50.122093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.122063 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c25f7963-5f1c-423b-a63c-6aee2121faad" containerName="maas-api" Oct 30 05:10:50.122278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.122103 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f7963-5f1c-423b-a63c-6aee2121faad" containerName="maas-api" Oct 30 05:10:50.122393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.122377 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25f7963-5f1c-423b-a63c-6aee2121faad" containerName="maas-api" Oct 30 05:10:50.142263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.129365 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.153390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.153357 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:10:50.210428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210377 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210440 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210612 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210717 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210792 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210906 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.210972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.210957 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.211039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.211019 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5rp\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.211123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.211108 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313000 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313148 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313178 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313276 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313321 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313383 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313459 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313537 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313572 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313604 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313642 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313652 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5rp\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.313979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313956 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.314042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.313984 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.315821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.315794 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.315930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.315846 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.334040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.334006 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.334280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.334261 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5rp\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.455484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.455452 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:50.781676 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:10:50.781643 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ed398e_0a28_4034_83dc_34734088711b.slice/crio-c294ad7861a62aac9ff8baecc42ac82704aaa72f7acad983ec36729ee85476b9 WatchSource:0}: Error finding container c294ad7861a62aac9ff8baecc42ac82704aaa72f7acad983ec36729ee85476b9: Status 404 returned error can't find the container with id c294ad7861a62aac9ff8baecc42ac82704aaa72f7acad983ec36729ee85476b9 Oct 30 05:10:50.784651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.784614 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:50.784734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.784686 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:50.784734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.784715 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:50.785122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:50.785094 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:10:51.572884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:51.572843 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" event={"ID":"a6ed398e-0a28-4034-83dc-34734088711b","Type":"ContainerStarted","Data":"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3"} Oct 30 05:10:51.573328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:51.572896 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" event={"ID":"a6ed398e-0a28-4034-83dc-34734088711b","Type":"ContainerStarted","Data":"c294ad7861a62aac9ff8baecc42ac82704aaa72f7acad983ec36729ee85476b9"} Oct 30 05:10:51.622279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:51.622197 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podStartSLOduration=1.6221801390000001 podStartE2EDuration="1.622180139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:51.618269089 +0000 UTC m=+1624.587764074" watchObservedRunningTime="2026-10-30 05:10:51.622180139 +0000 UTC m=+1624.591675126" Oct 30 05:10:52.456541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:52.456509 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:10:52.458283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:52.458258 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:52.458375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:52.458314 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:53.456407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:53.456375 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:53.456892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:53.456432 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:54.466441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:54.465880 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:54.466441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:54.465944 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:55.493246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:55.461879 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:55.493246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:55.461943 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:55.597279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:55.594087 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:55.597279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:55.594624 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 05:10:55.841926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:55.841811 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.001399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.001352 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.177466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.177412 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.272793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.272744 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.388801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.387606 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.466433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.466342 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:56.466680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.466647 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:56.477362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.477312 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.844455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.844337 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.845628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:56.845599 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:10:57.285278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.273342 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:10:57.363286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.285556 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 05:10:57.363554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.304401 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 05:10:57.565303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.542298 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": context deadline exceeded" Oct 30 05:10:57.606488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.588298 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.765271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.715910 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:57.820697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.820600 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:57.857785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.857535 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 05:10:57.862738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:57.734266 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.190048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.112419 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.628305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.557240 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.628305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.608766 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455" podUID="b292f152-c018-4589-a01b-e7f49c505f47" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.647473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.647413 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Oct 30 05:10:58.724473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.724423 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:10:58.902299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.879530 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 05:10:58.925282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:58.913446 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.062247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.044804 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:59.062247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.044863 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:59.166309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.135262 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 05:10:59.169657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.169629 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.169921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.169580 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.488306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.487686 5058 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:10:59.488306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.487752 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 05:10:59.570446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.570400 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.609195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.608355 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.673244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.658330 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.769396 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:10:59.762453 5058 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.589s" Oct 30 05:10:59.769396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.762540 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:10:59.769396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.762598 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:10:59.849534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.849493 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.880413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.880374 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.921425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.921373 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999804 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999865 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999899 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999932 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999962 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:10:59.999978 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.000010 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.000025 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:11:00.003290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.000052 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:11:00.116650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.116552 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:11:00.117341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.117316 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:11:00.157413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.157383 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:11:00.257243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:00.224189 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:11:01.272574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:01.272537 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:11:01.273784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:01.273724 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:11:01.282109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:01.282078 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:11:06.925077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:06.925030 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:11:06.931374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:06.931346 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:06.950801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:06.950772 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:11:06.962961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:06.962929 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b0814c-serving-cert\"" Oct 30 05:11:07.066946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.066906 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tr7c\" (UniqueName: \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.067121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.066953 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.067204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.067180 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.067361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.067338 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.175029 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.174989 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.175371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.175308 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.175815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.175792 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tr7c\" (UniqueName: \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.175937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.175923 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.202332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.199153 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.202332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.199981 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.202332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.201478 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.224276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.224180 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tr7c\" (UniqueName: \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\") pod \"maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.255485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.255151 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:07.594226 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:07.594181 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:11:08.496974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.496887 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" event={"ID":"c6f02ae7-e23f-441d-8abf-80afa1624f66","Type":"ContainerStarted","Data":"18ddb24f8dbd7d614a9961628e1402e234cdb2bde4b2044e3e063f5072b5519c"} Oct 30 05:11:08.497756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.497717 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" event={"ID":"c6f02ae7-e23f-441d-8abf-80afa1624f66","Type":"ContainerStarted","Data":"7a994183621698b881dbac92b72e8eec585d60e8b926c4a39a4fa0d2720be39f"} Oct 30 05:11:08.499697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.499661 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:08.503016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.502990 5058 generic.go:356] "Generic (PLEG): container finished" podID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerID="926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5" exitCode=0 Oct 30 05:11:08.503250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.503212 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerDied","Data":"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5"} Oct 30 05:11:08.503896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.503844 5058 scope.go:122] "RemoveContainer" containerID="926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5" Oct 30 05:11:08.507434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.507371 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:08.508131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.507946 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:08.508337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.508314 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:08.556403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:08.556346 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" podStartSLOduration=2.130147681 podStartE2EDuration="2.55633133s" totalImagesPullingTime="426.183649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:08.555432732 +0000 UTC m=+1641.524927741" watchObservedRunningTime="2026-10-30 05:11:08.55633133 +0000 UTC m=+1641.525826319" Oct 30 05:11:09.515764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:09.515730 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerStarted","Data":"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db"} Oct 30 05:11:09.529249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:09.524372 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:11:09.529249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:09.524576 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:09.529249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:09.524629 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:10.523787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:10.523757 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:10.524207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:10.523802 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:10.764407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:10.764363 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:11:10.789262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:10.789130 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:11:11.544239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:11.538429 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:11.544239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:11.538504 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:12.523809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:12.523776 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:12.524066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:12.523830 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:13.524039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:13.524003 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:13.524576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:13.524055 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:14.524082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:14.524050 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 05:11:14.524583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:14.524098 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 05:11:15.498393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.498362 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:11:15.528836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.528806 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:11:15.551533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.551456 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:11:15.675865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.675837 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:11:15.765412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.765318 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" podUID="09054a92-a469-42a5-a610-ce9b9042d503" containerName="payload-processing" containerID="cri-o://0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417" gracePeriod=25 Oct 30 05:11:15.789653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:15.789607 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" podUID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" containerName="payload-pre-processing" containerID="cri-o://f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff" gracePeriod=25 Oct 30 05:11:16.081726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.081696 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:11:16.107726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.107695 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\") pod \"09054a92-a469-42a5-a610-ce9b9042d503\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " Oct 30 05:11:16.107932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.107833 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\" (UniqueName: \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\") pod \"09054a92-a469-42a5-a610-ce9b9042d503\" (UID: \"09054a92-a469-42a5-a610-ce9b9042d503\") " Oct 30 05:11:16.108142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.108111 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume" pod "09054a92-a469-42a5-a610-ce9b9042d503" (UID: "09054a92-a469-42a5-a610-ce9b9042d503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:16.108686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.108650 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09054a92-a469-42a5-a610-ce9b9042d503-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:16.110416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.110389 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw" pod "09054a92-a469-42a5-a610-ce9b9042d503" (UID: "09054a92-a469-42a5-a610-ce9b9042d503"). InnerVolumeSpecName "kube-api-access-ffklw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:16.145818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.145790 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:11:16.209395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.209358 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\" (UniqueName: \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\") pod \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " Oct 30 05:11:16.209596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.209508 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\") pod \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\" (UID: \"6d58a1f9-0cf4-480f-8238-c60f9f3fd092\") " Oct 30 05:11:16.210164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.210122 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume" pod "6d58a1f9-0cf4-480f-8238-c60f9f3fd092" (UID: "6d58a1f9-0cf4-480f-8238-c60f9f3fd092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:16.210164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.210149 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffklw\" (UniqueName: \"kubernetes.io/projected/09054a92-a469-42a5-a610-ce9b9042d503-kube-api-access-ffklw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:16.212162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.212138 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c" pod "6d58a1f9-0cf4-480f-8238-c60f9f3fd092" (UID: "6d58a1f9-0cf4-480f-8238-c60f9f3fd092"). InnerVolumeSpecName "kube-api-access-qhc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:16.311279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.311172 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhc5c\" (UniqueName: \"kubernetes.io/projected/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-kube-api-access-qhc5c\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:16.311279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.311198 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d58a1f9-0cf4-480f-8238-c60f9f3fd092-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:16.683245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.683196 5058 generic.go:356] "Generic (PLEG): container finished" podID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" containerID="f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff" exitCode=0 Oct 30 05:11:16.683698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.683276 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" Oct 30 05:11:16.683698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.683284 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" event={"ID":"6d58a1f9-0cf4-480f-8238-c60f9f3fd092","Type":"ContainerDied","Data":"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff"} Oct 30 05:11:16.683698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.683336 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh" event={"ID":"6d58a1f9-0cf4-480f-8238-c60f9f3fd092","Type":"ContainerDied","Data":"8a14bc6bfe34d9a8a22ff3e804de3b8497a74a02be210f5ce97ea4420b83f4a4"} Oct 30 05:11:16.683698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.683360 5058 scope.go:122] "RemoveContainer" containerID="f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff" Oct 30 05:11:16.685045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.685011 5058 generic.go:356] "Generic (PLEG): container finished" podID="09054a92-a469-42a5-a610-ce9b9042d503" containerID="0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417" exitCode=0 Oct 30 05:11:16.685238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.685194 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" event={"ID":"09054a92-a469-42a5-a610-ce9b9042d503","Type":"ContainerDied","Data":"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417"} Oct 30 05:11:16.685339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.685255 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" event={"ID":"09054a92-a469-42a5-a610-ce9b9042d503","Type":"ContainerDied","Data":"0e719ae3e83f936214404179870b9412061c6ea412bbf342e436a4f772f740b2"} Oct 30 05:11:16.685407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.685377 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp" Oct 30 05:11:16.696082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.696065 5058 scope.go:122] "RemoveContainer" containerID="f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff" Oct 30 05:11:16.696437 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:16.696419 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff\": container with ID starting with f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff not found: ID does not exist" containerID="f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff" Oct 30 05:11:16.696503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.696442 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff"} err="failed to get container status \"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff\": rpc error: code = NotFound desc = could not find container \"f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff\": container with ID starting with f591abaaaacb25c61dff61aaa57f6cd7e8ad7f3f213c293cb5901216fedc00ff not found: ID does not exist" Oct 30 05:11:16.696503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.696457 5058 scope.go:122] "RemoveContainer" containerID="0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417" Oct 30 05:11:16.706525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.706502 5058 scope.go:122] "RemoveContainer" containerID="0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417" Oct 30 05:11:16.706837 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:16.706815 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417\": container with ID starting with 0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417 not found: ID does not exist" containerID="0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417" Oct 30 05:11:16.706898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.706849 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417"} err="failed to get container status \"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417\": rpc error: code = NotFound desc = could not find container \"0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417\": container with ID starting with 0cd428bbf4eaf3f5feb38aeda9990dff10c8c4b44789090aac509c4a19d3c417 not found: ID does not exist" Oct 30 05:11:16.739531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.739503 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:11:16.752620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.752593 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp"] Oct 30 05:11:16.850096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.850008 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:11:16.858453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:16.858313 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh"] Oct 30 05:11:17.163876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:17.163845 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09054a92-a469-42a5-a610-ce9b9042d503" path="/var/lib/kubelet/pods/09054a92-a469-42a5-a610-ce9b9042d503/volumes" Oct 30 05:11:17.164351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:17.164337 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" path="/var/lib/kubelet/pods/6d58a1f9-0cf4-480f-8238-c60f9f3fd092/volumes" Oct 30 05:11:19.191331 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:19.191295 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:11:29.857366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.857266 5058 generic.go:356] "Generic (PLEG): container finished" podID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerID="1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57" exitCode=0 Oct 30 05:11:29.857366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.857347 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerDied","Data":"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57"} Oct 30 05:11:29.857837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.857822 5058 scope.go:122] "RemoveContainer" containerID="1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57" Oct 30 05:11:29.859725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.859699 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:29.859795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.859765 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:29.859835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:29.859799 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:30.874263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:30.872849 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerStarted","Data":"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375"} Oct 30 05:11:31.220610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:31.220578 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:11:31.222313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:31.222231 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:31.222313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:31.222284 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:32.220021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:32.219987 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:32.220439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:32.220043 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:32.742378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:32.742335 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:11:33.200229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.200178 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:11:33.200691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.200660 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" containerID="cri-o://60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5" gracePeriod=30 Oct 30 05:11:33.210535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.207835 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:11:33.210535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.208042 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" containerID="cri-o://ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d" gracePeriod=30 Oct 30 05:11:33.221017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.220296 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:11:33.221017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.220582 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" containerID="cri-o://e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de" gracePeriod=30 Oct 30 05:11:33.222870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.222679 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:33.222870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.222738 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:33.260656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.260303 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:11:33.260656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:33.260551 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" containerID="cri-o://eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3" gracePeriod=30 Oct 30 05:11:34.225279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:34.224449 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:34.225279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:34.224547 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:35.220081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:35.220046 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:35.220304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:35.220096 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:36.220275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:36.220206 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 05:11:36.220683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:36.220323 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 05:11:37.225111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:37.225078 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:11:37.226138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:37.226114 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:11:38.586083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.586052 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:11:38.640231 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640187 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640276 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640335 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640402 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640442 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640560 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.640677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640628 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.641174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640768 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.641174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.640817 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\") pod \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\" (UID: \"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46\") " Oct 30 05:11:38.642679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.642637 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.642791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.642773 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.642961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.642878 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.642961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.642946 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.643982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.643960 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.644615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.644595 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "kube-api-access-ccvlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.647284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.647261 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:11:38.651666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.651630 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.651926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.651903 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.655036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.655014 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:11:38.655348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.655091 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo" pod "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" (UID: "f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.690683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.690655 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:11:38.742666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742629 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.742835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742692 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.742835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742737 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.742835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742758 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.742835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742796 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.742835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742822 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742861 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742908 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742945 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.742998 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743008 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743033 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.743112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743099 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.743357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743159 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.743357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743195 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.743357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743208 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.743357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743308 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.743357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743345 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743420 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743463 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743477 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743507 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743514 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743565 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743688 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743731 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743729 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743758 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743786 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743826 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743884 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\") pod \"d715f702-dbb5-4595-a978-152131f5da48\" (UID: \"d715f702-dbb5-4595-a978-152131f5da48\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743944 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\") pod \"a2295196-ca25-4d90-965a-9be42fd3b09b\" (UID: \"a2295196-ca25-4d90-965a-9be42fd3b09b\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743977 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\") pod \"a6ed398e-0a28-4034-83dc-34734088711b\" (UID: \"a6ed398e-0a28-4034-83dc-34734088711b\") " Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.743988 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.744122 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.744272 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.744777 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.744979 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745000 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745013 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2295196-ca25-4d90-965a-9be42fd3b09b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745026 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745039 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d715f702-dbb5-4595-a978-152131f5da48-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745055 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745069 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745082 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745095 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745108 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745120 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745133 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745146 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745158 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvlb\" (UniqueName: \"kubernetes.io/projected/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-kube-api-access-ccvlb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745171 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745184 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745196 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745212 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745242 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745283 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745388 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745399 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745456 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745585 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.746152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.745908 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.747190 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "kube-api-access-gj5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.747380 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.747416 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.747525 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.747876 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "kube-api-access-kck5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.748617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.748359 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.750039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.750011 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.750653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.750624 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "kube-api-access-7h2jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.750757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.750675 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo" pod "d715f702-dbb5-4595-a978-152131f5da48" (UID: "d715f702-dbb5-4595-a978-152131f5da48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.750835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.750814 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo" pod "a2295196-ca25-4d90-965a-9be42fd3b09b" (UID: "a2295196-ca25-4d90-965a-9be42fd3b09b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.751009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.750883 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy" pod "a6ed398e-0a28-4034-83dc-34734088711b" (UID: "a6ed398e-0a28-4034-83dc-34734088711b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.847117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847033 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847070 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847084 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847097 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847111 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ed398e-0a28-4034-83dc-34734088711b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847124 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847135 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847148 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847161 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d715f702-dbb5-4595-a978-152131f5da48-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847174 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kck5z\" (UniqueName: \"kubernetes.io/projected/d715f702-dbb5-4595-a978-152131f5da48-kube-api-access-kck5z\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847189 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ed398e-0a28-4034-83dc-34734088711b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847203 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847262 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2jx\" (UniqueName: \"kubernetes.io/projected/a2295196-ca25-4d90-965a-9be42fd3b09b-kube-api-access-7h2jx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847278 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5rp\" (UniqueName: \"kubernetes.io/projected/a6ed398e-0a28-4034-83dc-34734088711b-kube-api-access-gj5rp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847290 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ed398e-0a28-4034-83dc-34734088711b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847305 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d715f702-dbb5-4595-a978-152131f5da48-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.847395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.847318 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2295196-ca25-4d90-965a-9be42fd3b09b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.985431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.985396 5058 generic.go:356] "Generic (PLEG): container finished" podID="a6ed398e-0a28-4034-83dc-34734088711b" containerID="eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3" exitCode=0 Oct 30 05:11:38.985633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.985449 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" Oct 30 05:11:38.985633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.985497 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" event={"ID":"a6ed398e-0a28-4034-83dc-34734088711b","Type":"ContainerDied","Data":"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3"} Oct 30 05:11:38.985633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.985550 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q" event={"ID":"a6ed398e-0a28-4034-83dc-34734088711b","Type":"ContainerDied","Data":"c294ad7861a62aac9ff8baecc42ac82704aaa72f7acad983ec36729ee85476b9"} Oct 30 05:11:38.985633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.985587 5058 scope.go:122] "RemoveContainer" containerID="eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3" Oct 30 05:11:38.987158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.987134 5058 generic.go:356] "Generic (PLEG): container finished" podID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerID="60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5" exitCode=0 Oct 30 05:11:38.987300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.987191 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" Oct 30 05:11:38.987300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.987250 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" event={"ID":"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46","Type":"ContainerDied","Data":"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5"} Oct 30 05:11:38.987431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.987316 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t" event={"ID":"f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46","Type":"ContainerDied","Data":"171e6abe6bc1482c3d7283d00e4e16c74b41035aaad596fa06ee73ab840ce126"} Oct 30 05:11:38.994298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.988996 5058 generic.go:356] "Generic (PLEG): container finished" podID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerID="ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d" exitCode=0 Oct 30 05:11:38.994298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.989160 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" event={"ID":"a2295196-ca25-4d90-965a-9be42fd3b09b","Type":"ContainerDied","Data":"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d"} Oct 30 05:11:38.994298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.989200 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" event={"ID":"a2295196-ca25-4d90-965a-9be42fd3b09b","Type":"ContainerDied","Data":"2244ce60feac8254420e1d16a490de52fb698ed6a672301c26f09ad9b92ab99b"} Oct 30 05:11:38.994298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:38.989323 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv" Oct 30 05:11:39.007499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.007477 5058 scope.go:122] "RemoveContainer" containerID="eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3" Oct 30 05:11:39.007775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.007748 5058 generic.go:356] "Generic (PLEG): container finished" podID="d715f702-dbb5-4595-a978-152131f5da48" containerID="e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de" exitCode=0 Oct 30 05:11:39.007963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.007938 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" event={"ID":"d715f702-dbb5-4595-a978-152131f5da48","Type":"ContainerDied","Data":"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de"} Oct 30 05:11:39.008075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.007995 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" event={"ID":"d715f702-dbb5-4595-a978-152131f5da48","Type":"ContainerDied","Data":"62608cb300e9549f1c2082bb92ba691a37f3cc2c7551277980f8edd87dbd0f2b"} Oct 30 05:11:39.008075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.008065 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm" Oct 30 05:11:39.009152 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:39.009121 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3\": container with ID starting with eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3 not found: ID does not exist" containerID="eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3" Oct 30 05:11:39.009308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.009282 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3"} err="failed to get container status \"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3\": rpc error: code = NotFound desc = could not find container \"eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3\": container with ID starting with eb9e67ba8fb713de3e93f624bf927d74456463f0be3c98b075a36f59b30855e3 not found: ID does not exist" Oct 30 05:11:39.009541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.009525 5058 scope.go:122] "RemoveContainer" containerID="60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5" Oct 30 05:11:39.025063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.025039 5058 scope.go:122] "RemoveContainer" containerID="60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5" Oct 30 05:11:39.025445 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:39.025422 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5\": container with ID starting with 60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5 not found: ID does not exist" containerID="60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5" Oct 30 05:11:39.025511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.025455 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5"} err="failed to get container status \"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5\": rpc error: code = NotFound desc = could not find container \"60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5\": container with ID starting with 60124e6f1bcfff7b8745251a466e9035fe693bab0aa9ee26767b47400ee287f5 not found: ID does not exist" Oct 30 05:11:39.025511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.025480 5058 scope.go:122] "RemoveContainer" containerID="ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d" Oct 30 05:11:39.039535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.039484 5058 scope.go:122] "RemoveContainer" containerID="ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d" Oct 30 05:11:39.039927 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:39.039899 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d\": container with ID starting with ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d not found: ID does not exist" containerID="ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d" Oct 30 05:11:39.040028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.039942 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d"} err="failed to get container status \"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d\": rpc error: code = NotFound desc = could not find container \"ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d\": container with ID starting with ff384693d41191f68eff0952b36cbd4058ac4f6aa83b1680c4e6cd016404434d not found: ID does not exist" Oct 30 05:11:39.040028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.039968 5058 scope.go:122] "RemoveContainer" containerID="e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de" Oct 30 05:11:39.055023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.054998 5058 scope.go:122] "RemoveContainer" containerID="e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de" Oct 30 05:11:39.055409 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:39.055387 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de\": container with ID starting with e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de not found: ID does not exist" containerID="e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de" Oct 30 05:11:39.055514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.055421 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de"} err="failed to get container status \"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de\": rpc error: code = NotFound desc = could not find container \"e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de\": container with ID starting with e7b7556b47632ac4188ca04029c38e666a20d2500994c978009f6e7cd169d6de not found: ID does not exist" Oct 30 05:11:39.107202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.107125 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:11:39.111733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.111706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv"] Oct 30 05:11:39.146722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.146694 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:11:39.150370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.150349 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t"] Oct 30 05:11:39.163320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.163291 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" path="/var/lib/kubelet/pods/a2295196-ca25-4d90-965a-9be42fd3b09b/volumes" Oct 30 05:11:39.163847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.163830 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" path="/var/lib/kubelet/pods/f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46/volumes" Oct 30 05:11:39.186117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.186087 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:11:39.189881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.189858 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm"] Oct 30 05:11:39.226986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.226957 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:11:39.231812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:39.231787 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q"] Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.553971 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555339 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" containerName="payload-pre-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555375 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" containerName="payload-pre-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555422 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555429 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555444 5058 cpu_manager.go:414] "containerMap: removing container" podUID="09054a92-a469-42a5-a610-ce9b9042d503" containerName="payload-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555451 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09054a92-a469-42a5-a610-ce9b9042d503" containerName="payload-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555480 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555487 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555501 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555508 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555532 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555538 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555697 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="09054a92-a469-42a5-a610-ce9b9042d503" containerName="payload-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555712 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d715f702-dbb5-4595-a978-152131f5da48" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555720 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2295196-ca25-4d90-965a-9be42fd3b09b" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555733 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f1fd6f-0bee-4ad6-8b36-579d6eaecb46" containerName="istio-proxy" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555747 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d58a1f9-0cf4-480f-8238-c60f9f3fd092" containerName="payload-pre-processing" Oct 30 05:11:40.564244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.555763 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ed398e-0a28-4034-83dc-34734088711b" containerName="istio-proxy" Oct 30 05:11:40.581516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.581483 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.612156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.612095 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c7f962-openshift-default-dockercfg-vt6b4\"" Oct 30 05:11:40.613265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.613234 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.735985 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtpkh\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736119 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736156 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736194 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736328 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736396 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736438 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736460 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.737243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.736587 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.838197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.838114 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.838479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.838452 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.838625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.838609 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.838845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.838825 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839072 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtpkh\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839292 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.838625 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839139 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839564 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839680 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839796 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.839956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839939 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.840343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.839743 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.840646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.840624 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.841780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.841751 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.844957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.844916 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.865647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.865612 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:40.865770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:40.865758 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtpkh\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\") pod \"e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:41.128584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.128496 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:41.165666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.165627 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ed398e-0a28-4034-83dc-34734088711b" path="/var/lib/kubelet/pods/a6ed398e-0a28-4034-83dc-34734088711b/volumes" Oct 30 05:11:41.166390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.166367 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d715f702-dbb5-4595-a978-152131f5da48" path="/var/lib/kubelet/pods/d715f702-dbb5-4595-a978-152131f5da48/volumes" Oct 30 05:11:41.444054 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:11:41.444024 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97c5ba4_a3f6_47c6_8cc5_0442c2ae3e2a.slice/crio-20bc767b4ce891efd3b9ab8640f46d472433b83e72c1fc80a27297070acdff0f WatchSource:0}: Error finding container 20bc767b4ce891efd3b9ab8640f46d472433b83e72c1fc80a27297070acdff0f: Status 404 returned error can't find the container with id 20bc767b4ce891efd3b9ab8640f46d472433b83e72c1fc80a27297070acdff0f Oct 30 05:11:41.444875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.444854 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:11:41.446857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.446827 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:41.446904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.446891 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:41.446935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:41.446921 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:42.084530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.080376 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerStarted","Data":"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8"} Oct 30 05:11:42.084530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.080479 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerStarted","Data":"20bc767b4ce891efd3b9ab8640f46d472433b83e72c1fc80a27297070acdff0f"} Oct 30 05:11:42.133615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.133574 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:42.144795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.144749 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:11:42.145187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.145151 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:11:42.153329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:42.148832 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podStartSLOduration=2.148815876 podStartE2EDuration="2.148815876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:42.146287307 +0000 UTC m=+1675.115782294" watchObservedRunningTime="2026-10-30 05:11:42.148815876 +0000 UTC m=+1675.118310864" Oct 30 05:11:43.138854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:43.138524 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:44.096088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:44.096055 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:44.097189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:44.097165 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:11:46.985244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:46.976730 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:11:47.086423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.086328 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.151242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.145603 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c7f962-dockercfg-cdl5p\"" Oct 30 05:11:47.191246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.181811 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c7f962\"" Oct 30 05:11:47.191246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.183369 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:11:47.202266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.198633 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.202266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.198826 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b25r\" (UniqueName: \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.300881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.300794 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.301060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.300942 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b25r\" (UniqueName: \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.301996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.301964 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.323269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.323150 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b25r\" (UniqueName: \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\") pod \"payload-processing-e2e-inf-a-c7f962-6886679566-84khk\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:47.620886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.620796 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c7f962-dockercfg-cdl5p\"" Oct 30 05:11:47.636375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:47.636324 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:48.050331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:48.050297 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:11:48.310562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:48.310469 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" event={"ID":"9bf5cd1c-b395-4027-8cba-24c083467d2b","Type":"ContainerStarted","Data":"8c812df0c4a91df339b491fb24768d9fcb674fc4daeb92c24bcc130f1a84fa6a"} Oct 30 05:11:48.310562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:48.310523 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" event={"ID":"9bf5cd1c-b395-4027-8cba-24c083467d2b","Type":"ContainerStarted","Data":"00b317e4b01dc1b8540420dec3b9b2371018712477e69e294d4800dcbcd6dc59"} Oct 30 05:11:48.310765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:48.310678 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:48.372994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:48.372945 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" podStartSLOduration=2.372930953 podStartE2EDuration="2.372930953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:48.36855812 +0000 UTC m=+1681.338053108" watchObservedRunningTime="2026-10-30 05:11:48.372930953 +0000 UTC m=+1681.342425990" Oct 30 05:11:49.251263 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:11:49.250096 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:11:52.485361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:52.485323 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:11:52.485906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:52.485389 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:11:53.364838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.364804 5058 generic.go:356] "Generic (PLEG): container finished" podID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerID="40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db" exitCode=0 Oct 30 05:11:53.365059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.364875 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerDied","Data":"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db"} Oct 30 05:11:53.365377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.365361 5058 scope.go:122] "RemoveContainer" containerID="40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db" Oct 30 05:11:53.367132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.367105 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:53.367200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.367175 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:53.367252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:53.367206 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:54.375200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:54.375164 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerStarted","Data":"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292"} Oct 30 05:11:54.478936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:54.478897 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:11:54.479151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:54.479130 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:54.479205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:54.479178 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:11:55.480246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:55.479766 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:55.480246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:55.479824 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:11:56.479353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:56.478659 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:56.479353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:56.478739 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:11:57.471052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.471015 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:11:57.476269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.476213 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.478159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.478130 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:57.478304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.478186 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:11:57.509770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.509732 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f8eb4\"" Oct 30 05:11:57.509993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.509975 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f8eb4-dockercfg-68lmh\"" Oct 30 05:11:57.515151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.515117 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:11:57.818531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.817929 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.818531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.818166 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9tk\" (UniqueName: \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.919712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.919676 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.919885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.919847 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9tk\" (UniqueName: \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.920525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.920464 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:57.971250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:57.970288 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9tk\" (UniqueName: \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\") pod \"payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:58.092332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.092211 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:58.414849 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:11:58.414801 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f9a72e_f69c_4eaf_9a58_906d2aae2e3f.slice/crio-4a36edb52c8543d657de8f75af21c705b112459dfedeb367d36227f06fb97970 WatchSource:0}: Error finding container 4a36edb52c8543d657de8f75af21c705b112459dfedeb367d36227f06fb97970: Status 404 returned error can't find the container with id 4a36edb52c8543d657de8f75af21c705b112459dfedeb367d36227f06fb97970 Oct 30 05:11:58.416093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.416070 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:11:58.478794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.478171 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:58.478794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.478243 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:11:58.864381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.864273 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" event={"ID":"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f","Type":"ContainerStarted","Data":"412952d75c97fc2bcf82ecd332c23208c8651582806203904dc50c1cf5543e10"} Oct 30 05:11:58.864381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.864350 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" event={"ID":"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f","Type":"ContainerStarted","Data":"4a36edb52c8543d657de8f75af21c705b112459dfedeb367d36227f06fb97970"} Oct 30 05:11:58.864381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.864379 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:11:58.914016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:58.913950 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" podStartSLOduration=1.913935794 podStartE2EDuration="1.913935794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:58.913784857 +0000 UTC m=+1691.883279846" watchObservedRunningTime="2026-10-30 05:11:58.913935794 +0000 UTC m=+1691.883430781" Oct 30 05:11:59.322713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:59.322683 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:11:59.478402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:59.478367 5058 patch_prober.go:29] interesting pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:11:59.478581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:11:59.478434 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:12:00.482630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:00.482596 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:12:00.483074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:00.483046 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:12:00.483804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:00.483782 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:12:01.468613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.468460 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:12:01.477303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.477262 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.514792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.514762 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c3486b-dockercfg-q7h24\"" Oct 30 05:12:01.521553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.521523 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c3486b\"" Oct 30 05:12:01.535316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.535281 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:12:01.554036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.541558 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.554036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.544148 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smsbv\" (UniqueName: \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.554036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.544311 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.572262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.562059 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:12:01.679265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.673805 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:12:01.810071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.797010 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.810071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.797209 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smsbv\" (UniqueName: \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.810071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.797315 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8m6\" (UniqueName: \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.810071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.797473 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.810071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.798806 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.824280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.823115 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smsbv\" (UniqueName: \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\") pod \"payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.824280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.823433 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:01.903260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.899114 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.903260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.900108 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.903260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.901006 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8m6\" (UniqueName: \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:01.927045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:01.927012 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8m6\" (UniqueName: \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\") pod \"payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:02.134112 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:02.134068 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b03b41_2e1b_48cf_8540_75b351ffaafb.slice/crio-d0a41a802ffa154313bfeca50ecb447cf8a1c49d1635cd4fe012a338ef07117a WatchSource:0}: Error finding container d0a41a802ffa154313bfeca50ecb447cf8a1c49d1635cd4fe012a338ef07117a: Status 404 returned error can't find the container with id d0a41a802ffa154313bfeca50ecb447cf8a1c49d1635cd4fe012a338ef07117a Oct 30 05:12:02.135289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.135267 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:12:02.190689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.190664 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:02.500911 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:02.500874 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5195b4bc_acd4_42cc_9cda_d6d2c2458785.slice/crio-d83c462bf62b9886afc9982a0794bdea0b89deff59e3d5db327d1f1333de7e78 WatchSource:0}: Error finding container d83c462bf62b9886afc9982a0794bdea0b89deff59e3d5db327d1f1333de7e78: Status 404 returned error can't find the container with id d83c462bf62b9886afc9982a0794bdea0b89deff59e3d5db327d1f1333de7e78 Oct 30 05:12:02.512238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.504133 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:12:02.917040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.916989 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" event={"ID":"5195b4bc-acd4-42cc-9cda-d6d2c2458785","Type":"ContainerStarted","Data":"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8"} Oct 30 05:12:02.917553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.917069 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" event={"ID":"5195b4bc-acd4-42cc-9cda-d6d2c2458785","Type":"ContainerStarted","Data":"d83c462bf62b9886afc9982a0794bdea0b89deff59e3d5db327d1f1333de7e78"} Oct 30 05:12:02.917553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.917100 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:02.918923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.918897 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" event={"ID":"36b03b41-2e1b-48cf-8540-75b351ffaafb","Type":"ContainerStarted","Data":"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8"} Oct 30 05:12:02.919031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.918944 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" event={"ID":"36b03b41-2e1b-48cf-8540-75b351ffaafb","Type":"ContainerStarted","Data":"d0a41a802ffa154313bfeca50ecb447cf8a1c49d1635cd4fe012a338ef07117a"} Oct 30 05:12:02.919031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.918999 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:02.973438 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:02.973382 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" podStartSLOduration=1.973367499 podStartE2EDuration="1.973367499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:02.972353375 +0000 UTC m=+1695.941848362" watchObservedRunningTime="2026-10-30 05:12:02.973367499 +0000 UTC m=+1695.942862487" Oct 30 05:12:03.029411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:03.029353 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" podStartSLOduration=2.029338792 podStartE2EDuration="2.029338792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:03.026084482 +0000 UTC m=+1695.995579469" watchObservedRunningTime="2026-10-30 05:12:03.029338792 +0000 UTC m=+1695.998833810" Oct 30 05:12:09.888150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:09.888117 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:12:13.930119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:13.930088 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:12:13.930566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:13.930155 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:12:18.359778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:18.359739 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:12:18.360400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:18.359984 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" podUID="3445476a-8e09-4468-864c-8be7aaa526e9" containerName="maas-api" containerID="cri-o://4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854" gracePeriod=30 Oct 30 05:12:18.827336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:18.826780 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:12:19.017041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.016952 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\") pod \"3445476a-8e09-4468-864c-8be7aaa526e9\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " Oct 30 05:12:19.025254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.017295 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\" (UniqueName: \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\") pod \"3445476a-8e09-4468-864c-8be7aaa526e9\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " Oct 30 05:12:19.025653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.025616 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") pod \"3445476a-8e09-4468-864c-8be7aaa526e9\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " Oct 30 05:12:19.069875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.069844 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\") pod \"3445476a-8e09-4468-864c-8be7aaa526e9\" (UID: \"3445476a-8e09-4468-864c-8be7aaa526e9\") " Oct 30 05:12:19.082245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.080518 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca" pod "3445476a-8e09-4468-864c-8be7aaa526e9" (UID: "3445476a-8e09-4468-864c-8be7aaa526e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:19.083299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.083275 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3445476a-8e09-4468-864c-8be7aaa526e9-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:19.083798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.083755 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls" pod "3445476a-8e09-4468-864c-8be7aaa526e9" (UID: "3445476a-8e09-4468-864c-8be7aaa526e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:19.083919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.083887 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc" pod "3445476a-8e09-4468-864c-8be7aaa526e9" (UID: "3445476a-8e09-4468-864c-8be7aaa526e9"). InnerVolumeSpecName "kube-api-access-ps9lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:19.090783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.090747 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs" pod "3445476a-8e09-4468-864c-8be7aaa526e9" (UID: "3445476a-8e09-4468-864c-8be7aaa526e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:19.097381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.097084 5058 generic.go:356] "Generic (PLEG): container finished" podID="3445476a-8e09-4468-864c-8be7aaa526e9" containerID="4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854" exitCode=0 Oct 30 05:12:19.097381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.097131 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" event={"ID":"3445476a-8e09-4468-864c-8be7aaa526e9","Type":"ContainerDied","Data":"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854"} Oct 30 05:12:19.097381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.097179 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" event={"ID":"3445476a-8e09-4468-864c-8be7aaa526e9","Type":"ContainerDied","Data":"d7525a2ff3afc26e5662a558ad5e5c50f1b25ce0e9e8c4722ff970f0716ca543"} Oct 30 05:12:19.097381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.097209 5058 scope.go:122] "RemoveContainer" containerID="4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854" Oct 30 05:12:19.097381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.097380 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh" Oct 30 05:12:19.124695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.124375 5058 scope.go:122] "RemoveContainer" containerID="4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854" Oct 30 05:12:19.125023 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:19.124930 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854\": container with ID starting with 4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854 not found: ID does not exist" containerID="4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854" Oct 30 05:12:19.125023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.124964 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854"} err="failed to get container status \"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854\": rpc error: code = NotFound desc = could not find container \"4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854\": container with ID starting with 4bc4c881670e8ac2df954637466180f6b89074abc1a6e7862516089111994854 not found: ID does not exist" Oct 30 05:12:19.178371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.178335 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:12:19.185598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.185565 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:19.185598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.185600 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9lc\" (UniqueName: \"kubernetes.io/projected/3445476a-8e09-4468-864c-8be7aaa526e9-kube-api-access-ps9lc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:19.185771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.185615 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3445476a-8e09-4468-864c-8be7aaa526e9-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:19.189885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:19.189859 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-543d08-6f787759db-nsvhh"] Oct 30 05:12:19.197953 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:19.197927 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:12:21.016768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.016736 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:12:21.018460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.016996 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" podUID="29286179-b14d-4e48-b94d-b5f74c473ff0" containerName="maas-api" containerID="cri-o://2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a" gracePeriod=30 Oct 30 05:12:21.168979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.168944 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3445476a-8e09-4468-864c-8be7aaa526e9" path="/var/lib/kubelet/pods/3445476a-8e09-4468-864c-8be7aaa526e9/volumes" Oct 30 05:12:21.474373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.474344 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:12:21.729331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.697521 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\") pod \"29286179-b14d-4e48-b94d-b5f74c473ff0\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " Oct 30 05:12:21.729331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.697608 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\") pod \"29286179-b14d-4e48-b94d-b5f74c473ff0\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " Oct 30 05:12:21.729331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.697717 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\") pod \"29286179-b14d-4e48-b94d-b5f74c473ff0\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " Oct 30 05:12:21.729331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.698005 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\" (UniqueName: \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\") pod \"29286179-b14d-4e48-b94d-b5f74c473ff0\" (UID: \"29286179-b14d-4e48-b94d-b5f74c473ff0\") " Oct 30 05:12:21.757640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.757605 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca" pod "29286179-b14d-4e48-b94d-b5f74c473ff0" (UID: "29286179-b14d-4e48-b94d-b5f74c473ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:21.799575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.796656 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls" pod "29286179-b14d-4e48-b94d-b5f74c473ff0" (UID: "29286179-b14d-4e48-b94d-b5f74c473ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:21.799575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.796799 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t" pod "29286179-b14d-4e48-b94d-b5f74c473ff0" (UID: "29286179-b14d-4e48-b94d-b5f74c473ff0"). InnerVolumeSpecName "kube-api-access-g6m8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:21.807755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.807714 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs" pod "29286179-b14d-4e48-b94d-b5f74c473ff0" (UID: "29286179-b14d-4e48-b94d-b5f74c473ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:21.808281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.808026 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:21.808281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.808056 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6m8t\" (UniqueName: \"kubernetes.io/projected/29286179-b14d-4e48-b94d-b5f74c473ff0-kube-api-access-g6m8t\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:21.808281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.808073 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29286179-b14d-4e48-b94d-b5f74c473ff0-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:21.808281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:21.808087 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29286179-b14d-4e48-b94d-b5f74c473ff0-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:22.151861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.151820 5058 generic.go:356] "Generic (PLEG): container finished" podID="29286179-b14d-4e48-b94d-b5f74c473ff0" containerID="2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a" exitCode=0 Oct 30 05:12:22.152689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.151884 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" Oct 30 05:12:22.152689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.151963 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" event={"ID":"29286179-b14d-4e48-b94d-b5f74c473ff0","Type":"ContainerDied","Data":"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a"} Oct 30 05:12:22.152689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.152023 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt" event={"ID":"29286179-b14d-4e48-b94d-b5f74c473ff0","Type":"ContainerDied","Data":"741f014c2c0a9d7ea19b398f551881db64104ee8876b681749a35582b933e916"} Oct 30 05:12:22.152689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.152052 5058 scope.go:122] "RemoveContainer" containerID="2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a" Oct 30 05:12:22.166424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.166404 5058 scope.go:122] "RemoveContainer" containerID="2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a" Oct 30 05:12:22.166762 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:22.166742 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a\": container with ID starting with 2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a not found: ID does not exist" containerID="2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a" Oct 30 05:12:22.166827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.166773 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a"} err="failed to get container status \"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a\": rpc error: code = NotFound desc = could not find container \"2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a\": container with ID starting with 2b1b9e81d25a2ba475bebd4daa26bd8b3d25d2330f1d44344a9d167f4ed5f61a not found: ID does not exist" Oct 30 05:12:22.243852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.243818 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:12:22.274238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:22.271030 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-645aef-fd4bc9976-v9fkt"] Oct 30 05:12:23.163868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:23.163836 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29286179-b14d-4e48-b94d-b5f74c473ff0" path="/var/lib/kubelet/pods/29286179-b14d-4e48-b94d-b5f74c473ff0/volumes" Oct 30 05:12:26.283497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.283456 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:12:26.285492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.285465 5058 cpu_manager.go:414] "containerMap: removing container" podUID="29286179-b14d-4e48-b94d-b5f74c473ff0" containerName="maas-api" Oct 30 05:12:26.285492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.285491 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29286179-b14d-4e48-b94d-b5f74c473ff0" containerName="maas-api" Oct 30 05:12:26.285682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.285580 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3445476a-8e09-4468-864c-8be7aaa526e9" containerName="maas-api" Oct 30 05:12:26.285682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.285588 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3445476a-8e09-4468-864c-8be7aaa526e9" containerName="maas-api" Oct 30 05:12:26.285986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.285965 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="29286179-b14d-4e48-b94d-b5f74c473ff0" containerName="maas-api" Oct 30 05:12:26.286047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.286007 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3445476a-8e09-4468-864c-8be7aaa526e9" containerName="maas-api" Oct 30 05:12:26.298861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.298824 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.315210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.315181 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:12:26.319013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.318988 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c7f962-serving-cert\"" Oct 30 05:12:26.406432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.406365 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.406876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.406852 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.407090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.407075 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94p9\" (UniqueName: \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.407310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.407293 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.512180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.512142 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.512458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.512212 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94p9\" (UniqueName: \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.512458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.512303 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.512587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.512481 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.514094 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.514065 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.515765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.515738 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.517777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.517755 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.542642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.542578 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94p9\" (UniqueName: \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\") pod \"maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.629815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.629782 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:26.947195 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:26.947165 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59235d6_516a_435b_93f8_16dbc4583ce7.slice/crio-4bb55e38c75286aa1437ce51ef298620d730b14cb92d311217be40f782b5b280 WatchSource:0}: Error finding container 4bb55e38c75286aa1437ce51ef298620d730b14cb92d311217be40f782b5b280: Status 404 returned error can't find the container with id 4bb55e38c75286aa1437ce51ef298620d730b14cb92d311217be40f782b5b280 Oct 30 05:12:26.950174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:26.950155 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:12:27.192293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:27.192254 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" event={"ID":"f59235d6-516a-435b-93f8-16dbc4583ce7","Type":"ContainerStarted","Data":"4bb55e38c75286aa1437ce51ef298620d730b14cb92d311217be40f782b5b280"} Oct 30 05:12:28.201564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:28.201532 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" event={"ID":"f59235d6-516a-435b-93f8-16dbc4583ce7","Type":"ContainerStarted","Data":"589a3ca1466d3ac2d52cd798b483e443d8f127a8a095f55b47aa86deef8ec5ec"} Oct 30 05:12:28.201988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:28.201619 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:28.271069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:28.271017 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" podStartSLOduration=1.8663373 podStartE2EDuration="2.271002217s" totalImagesPullingTime="404.664917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:28.267381744 +0000 UTC m=+1721.236876731" watchObservedRunningTime="2026-10-30 05:12:28.271002217 +0000 UTC m=+1721.240497205" Oct 30 05:12:34.215008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:34.214975 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:12:34.676583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:34.676549 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:12:34.677043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:34.676996 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerName="maas-api" containerID="cri-o://b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65" gracePeriod=30 Oct 30 05:12:34.690131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:34.690086 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.125:8443/health\": dial tcp 10.132.0.125:8443: connect: connection refused" Oct 30 05:12:35.103050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.103019 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:12:35.225024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.223449 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\") pod \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " Oct 30 05:12:35.225024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.223661 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\" (UniqueName: \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\") pod \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " Oct 30 05:12:35.225024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.223798 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\") pod \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " Oct 30 05:12:35.225024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.223898 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\") pod \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\" (UID: \"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6\") " Oct 30 05:12:35.226526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.226491 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca" pod "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" (UID: "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:35.236275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.231405 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt" pod "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" (UID: "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6"). InnerVolumeSpecName "kube-api-access-qxpzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:35.481104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.407698 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.481104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.407734 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpzt\" (UniqueName: \"kubernetes.io/projected/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-kube-api-access-qxpzt\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.481104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.480512 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls" pod "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" (UID: "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:35.509673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.509602 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.510089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.510063 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs" pod "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" (UID: "569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:35.537485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.534054 5058 generic.go:356] "Generic (PLEG): container finished" podID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerID="b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65" exitCode=0 Oct 30 05:12:35.537485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.534113 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" event={"ID":"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6","Type":"ContainerDied","Data":"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65"} Oct 30 05:12:35.537485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.534161 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" event={"ID":"569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6","Type":"ContainerDied","Data":"26aa6be22cf482781b9c882a122c2d72705abe7dca8d18de0b3d815e7709e257"} Oct 30 05:12:35.537485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.534191 5058 scope.go:122] "RemoveContainer" containerID="b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65" Oct 30 05:12:35.537485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.534391 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v" Oct 30 05:12:35.576040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.576011 5058 scope.go:122] "RemoveContainer" containerID="b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65" Oct 30 05:12:35.576504 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:35.576436 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65\": container with ID starting with b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65 not found: ID does not exist" containerID="b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65" Oct 30 05:12:35.576504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.576468 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65"} err="failed to get container status \"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65\": rpc error: code = NotFound desc = could not find container \"b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65\": container with ID starting with b38d64a1e3a08b14c40d6876bc3b1f914036e037cf0758cd3fea06fd5ef7ba65 not found: ID does not exist" Oct 30 05:12:35.611196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.611170 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.617112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.617090 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:12:35.629734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:35.629706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288e01-7f5f59df7d-8vv9v"] Oct 30 05:12:37.162737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:37.162701 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" path="/var/lib/kubelet/pods/569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6/volumes" Oct 30 05:12:38.074663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.074630 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:12:38.075486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.075469 5058 cpu_manager.go:414] "containerMap: removing container" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerName="maas-api" Oct 30 05:12:38.075542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.075487 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerName="maas-api" Oct 30 05:12:38.075613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.075604 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c0fb2-7fb2-4eda-bd4e-e9f849fd99d6" containerName="maas-api" Oct 30 05:12:38.079431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.079415 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.100920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.100888 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:12:38.109403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.109307 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-010013\"" Oct 30 05:12:38.117036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.112753 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-010013-dockercfg-mz6mg\"" Oct 30 05:12:38.261242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.260965 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdsl\" (UniqueName: \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.261242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.261033 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.396239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.394774 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdsl\" (UniqueName: \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.396239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.394831 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.396239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.395872 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.427267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.422532 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdsl\" (UniqueName: \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\") pod \"payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.445258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.443469 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:12:38.449714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.448996 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.453840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.451697 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:12:38.598514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.598479 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lqw\" (UniqueName: \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.598721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.598693 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.693750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.693656 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:38.700412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.700382 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lqw\" (UniqueName: \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.700606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.700590 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.701267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.701248 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.723931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.723902 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lqw\" (UniqueName: \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\") pod \"payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:38.774570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:38.774252 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:39.000152 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:39.000111 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a85529_1cd0_42f4_be40_edefe3420042.slice/crio-f4809bc0c6c1ad1c86c697ae5cd03b6a20a202e40fc033542d657db8db0986e9 WatchSource:0}: Error finding container f4809bc0c6c1ad1c86c697ae5cd03b6a20a202e40fc033542d657db8db0986e9: Status 404 returned error can't find the container with id f4809bc0c6c1ad1c86c697ae5cd03b6a20a202e40fc033542d657db8db0986e9 Oct 30 05:12:39.001799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.001777 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:12:39.087506 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:39.087480 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0a6347_86f9_4734_adb7_71037d2b38cb.slice/crio-070d4a36ffe0259a0ca262cde960355f7f6e875d3a6750cb5253beb53ac533b9 WatchSource:0}: Error finding container 070d4a36ffe0259a0ca262cde960355f7f6e875d3a6750cb5253beb53ac533b9: Status 404 returned error can't find the container with id 070d4a36ffe0259a0ca262cde960355f7f6e875d3a6750cb5253beb53ac533b9 Oct 30 05:12:39.088783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.088760 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:12:39.566356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.566310 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" event={"ID":"88a85529-1cd0-42f4-be40-edefe3420042","Type":"ContainerStarted","Data":"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b"} Oct 30 05:12:39.566833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.566379 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" event={"ID":"88a85529-1cd0-42f4-be40-edefe3420042","Type":"ContainerStarted","Data":"f4809bc0c6c1ad1c86c697ae5cd03b6a20a202e40fc033542d657db8db0986e9"} Oct 30 05:12:39.566833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.566406 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:39.568030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.568005 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" event={"ID":"5d0a6347-86f9-4734-adb7-71037d2b38cb","Type":"ContainerStarted","Data":"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60"} Oct 30 05:12:39.568160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.568048 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" event={"ID":"5d0a6347-86f9-4734-adb7-71037d2b38cb","Type":"ContainerStarted","Data":"070d4a36ffe0259a0ca262cde960355f7f6e875d3a6750cb5253beb53ac533b9"} Oct 30 05:12:39.568160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.568074 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:39.615505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.615450 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" podStartSLOduration=1.6154316180000001 podStartE2EDuration="1.615431618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:39.615062922 +0000 UTC m=+1732.584557910" watchObservedRunningTime="2026-10-30 05:12:39.615431618 +0000 UTC m=+1732.584926608" Oct 30 05:12:39.659050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.658999 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" podStartSLOduration=1.658981979 podStartE2EDuration="1.658981979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:39.657663015 +0000 UTC m=+1732.627158210" watchObservedRunningTime="2026-10-30 05:12:39.658981979 +0000 UTC m=+1732.628476968" Oct 30 05:12:39.668084 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.668052 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:12:39.673008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.672980 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.689512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.689278 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f8eb4-serving-cert\"" Oct 30 05:12:39.691430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.691407 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:12:39.712702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.712667 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpmj\" (UniqueName: \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.712932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.712724 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.713175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.713138 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.713282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.713265 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.815211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.815174 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.815453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.815429 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.815589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.815523 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.815655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.815642 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpmj\" (UniqueName: \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.816179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.816154 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.817799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.817732 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.817917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.817894 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:39.836956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:39.836932 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpmj\" (UniqueName: \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\") pod \"maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:40.004242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:40.001908 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:40.333141 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:40.333113 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44169c0_3ba7_4d92_b0b0_ddeedace2887.slice/crio-b974b98bce05b9f197c463e1a99776a8b95f0e3e437b3ccbd5be666bbde71921 WatchSource:0}: Error finding container b974b98bce05b9f197c463e1a99776a8b95f0e3e437b3ccbd5be666bbde71921: Status 404 returned error can't find the container with id b974b98bce05b9f197c463e1a99776a8b95f0e3e437b3ccbd5be666bbde71921 Oct 30 05:12:40.334697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:40.334673 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:12:40.577526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:40.577485 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" event={"ID":"a44169c0-3ba7-4d92-b0b0-ddeedace2887","Type":"ContainerStarted","Data":"b974b98bce05b9f197c463e1a99776a8b95f0e3e437b3ccbd5be666bbde71921"} Oct 30 05:12:41.068389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:41.068349 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:12:41.587711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:41.587667 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" event={"ID":"a44169c0-3ba7-4d92-b0b0-ddeedace2887","Type":"ContainerStarted","Data":"986175b0702d21b82ddef8ef76b3ff4ac42609cd3d62a4fc7d8ac16e05bc3ef9"} Oct 30 05:12:41.588193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:41.587754 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:41.638069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:41.637999 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" podStartSLOduration=2.334755831 podStartE2EDuration="2.637979497s" totalImagesPullingTime="303.223666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:41.633239835 +0000 UTC m=+1734.602734818" watchObservedRunningTime="2026-10-30 05:12:41.637979497 +0000 UTC m=+1734.607474487" Oct 30 05:12:43.942751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:43.937668 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:12:43.976366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:43.975641 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:12:43.976366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:43.975770 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:43.994609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:43.994083 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c3486b-serving-cert\"" Oct 30 05:12:44.142964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.142930 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.143113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.143072 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.143177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.143155 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7892\" (UniqueName: \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.143258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.143206 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.246241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.245237 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7892\" (UniqueName: \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.246241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.245331 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.246241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.245555 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.246241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.245663 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.256082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.256044 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.308209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.303181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.308209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.304977 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7892\" (UniqueName: \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.308209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.306850 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\") pod \"maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.379163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.379130 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:44.695479 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:12:44.695449 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c88fae8_438f_4e38_8c3f_58766f21e5e0.slice/crio-94d1b7fe3bc78639be8ce497175fc62f772e332e817fc090668b9f7fe7c47fee WatchSource:0}: Error finding container 94d1b7fe3bc78639be8ce497175fc62f772e332e817fc090668b9f7fe7c47fee: Status 404 returned error can't find the container with id 94d1b7fe3bc78639be8ce497175fc62f772e332e817fc090668b9f7fe7c47fee Oct 30 05:12:44.697378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.697354 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:12:44.765443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:44.765409 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:12:45.619984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:45.619950 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" event={"ID":"2c88fae8-438f-4e38-8c3f-58766f21e5e0","Type":"ContainerStarted","Data":"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f"} Oct 30 05:12:45.620487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:45.620036 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" event={"ID":"2c88fae8-438f-4e38-8c3f-58766f21e5e0","Type":"ContainerStarted","Data":"94d1b7fe3bc78639be8ce497175fc62f772e332e817fc090668b9f7fe7c47fee"} Oct 30 05:12:45.620487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:45.620189 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:45.667324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:45.667271 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" podStartSLOduration=2.290904913 podStartE2EDuration="2.667257776s" totalImagesPullingTime="376.352863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:45.665366897 +0000 UTC m=+1738.634861885" watchObservedRunningTime="2026-10-30 05:12:45.667257776 +0000 UTC m=+1738.636752763" Oct 30 05:12:46.068793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.068753 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" podUID="7808cbe3-5187-4e5f-9a32-d233eff308fb" containerName="payload-pre-processing" containerID="cri-o://c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2" gracePeriod=25 Oct 30 05:12:46.371913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.371888 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:12:46.463868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.446289 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\") pod \"7808cbe3-5187-4e5f-9a32-d233eff308fb\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " Oct 30 05:12:46.463868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.446380 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\" (UniqueName: \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\") pod \"7808cbe3-5187-4e5f-9a32-d233eff308fb\" (UID: \"7808cbe3-5187-4e5f-9a32-d233eff308fb\") " Oct 30 05:12:46.463868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.448101 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume" pod "7808cbe3-5187-4e5f-9a32-d233eff308fb" (UID: "7808cbe3-5187-4e5f-9a32-d233eff308fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:46.463868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.451041 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl" pod "7808cbe3-5187-4e5f-9a32-d233eff308fb" (UID: "7808cbe3-5187-4e5f-9a32-d233eff308fb"). InnerVolumeSpecName "kube-api-access-frkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:46.548728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.548690 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808cbe3-5187-4e5f-9a32-d233eff308fb-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:46.548728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.548727 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frkjl\" (UniqueName: \"kubernetes.io/projected/7808cbe3-5187-4e5f-9a32-d233eff308fb-kube-api-access-frkjl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:12:46.629692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.629607 5058 generic.go:356] "Generic (PLEG): container finished" podID="7808cbe3-5187-4e5f-9a32-d233eff308fb" containerID="c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2" exitCode=0 Oct 30 05:12:46.629692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.629658 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" Oct 30 05:12:46.630159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.629691 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" event={"ID":"7808cbe3-5187-4e5f-9a32-d233eff308fb","Type":"ContainerDied","Data":"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2"} Oct 30 05:12:46.630159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.629744 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2" event={"ID":"7808cbe3-5187-4e5f-9a32-d233eff308fb","Type":"ContainerDied","Data":"2d6aaac3a00ac4367d5ec9480fd8d68acadc36518e17014b24fbc8081fa8c7ea"} Oct 30 05:12:46.630159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.629766 5058 scope.go:122] "RemoveContainer" containerID="c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2" Oct 30 05:12:46.640937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.640917 5058 scope.go:122] "RemoveContainer" containerID="c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2" Oct 30 05:12:46.641266 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:46.641248 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2\": container with ID starting with c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2 not found: ID does not exist" containerID="c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2" Oct 30 05:12:46.641346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.641273 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2"} err="failed to get container status \"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2\": rpc error: code = NotFound desc = could not find container \"c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2\": container with ID starting with c5f4b6f232d2852df31cf71ffea82cd9835985bffcfabc731eac95ed144efae2 not found: ID does not exist" Oct 30 05:12:46.687892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.687856 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:12:46.701036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:46.701007 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2"] Oct 30 05:12:47.163768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:47.163736 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7808cbe3-5187-4e5f-9a32-d233eff308fb" path="/var/lib/kubelet/pods/7808cbe3-5187-4e5f-9a32-d233eff308fb/volumes" Oct 30 05:12:47.601544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:47.601472 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:12:49.201382 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:12:49.201344 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:12:50.580416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:50.580391 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:12:50.580820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:50.580455 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:12:51.636521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:51.636493 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:12:54.693109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:54.693070 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:12:54.723999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:54.723395 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:12:59.694198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:59.694093 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" podUID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" containerName="payload-processing" containerID="cri-o://33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a" gracePeriod=25 Oct 30 05:12:59.724647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:59.724602 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" podUID="29e51776-a819-4fe1-891e-522f0dd8b946" containerName="payload-pre-processing" containerID="cri-o://9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee" gracePeriod=25 Oct 30 05:12:59.787675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:12:59.787617 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" containerID="cri-o://95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958" gracePeriod=45 Oct 30 05:13:00.120300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.119780 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:13:00.135991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.135965 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:13:00.188791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.188759 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\" (UniqueName: \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\") pod \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " Oct 30 05:13:00.189188 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.189167 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\") pod \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\" (UID: \"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59\") " Oct 30 05:13:00.189339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.189289 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:13:00.189840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.189819 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume" pod "e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" (UID: "e3bfa743-8e65-47b2-90ba-bf7c4eb99f59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:00.190257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.190238 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.191773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.191749 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8" pod "e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" (UID: "e3bfa743-8e65-47b2-90ba-bf7c4eb99f59"). InnerVolumeSpecName "kube-api-access-r6gp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:00.291349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291253 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291344 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\" (UniqueName: \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291379 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291409 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291505 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291579 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\" (UniqueName: \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\") pod \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\" (UID: \"01f3acba-c988-41b2-bbd5-6fc61e1d8323\") " Oct 30 05:13:00.291812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291586 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:00.291812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291632 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\" (UniqueName: \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\") pod \"29e51776-a819-4fe1-891e-522f0dd8b946\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " Oct 30 05:13:00.291812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291662 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:00.291812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291757 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\") pod \"29e51776-a819-4fe1-891e-522f0dd8b946\" (UID: \"29e51776-a819-4fe1-891e-522f0dd8b946\") " Oct 30 05:13:00.292075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.291821 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:00.292547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.292523 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume" pod "29e51776-a819-4fe1-891e-522f0dd8b946" (UID: "29e51776-a819-4fe1-891e-522f0dd8b946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:00.292669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.292531 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.292777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.292763 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.292873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.292862 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.292957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.292947 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6gp8\" (UniqueName: \"kubernetes.io/projected/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59-kube-api-access-r6gp8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.294735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.294708 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "kube-api-access-7dhp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:00.296137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.296112 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:00.296137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.296129 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs" pod "01f3acba-c988-41b2-bbd5-6fc61e1d8323" (UID: "01f3acba-c988-41b2-bbd5-6fc61e1d8323"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:00.296428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.296402 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm" pod "29e51776-a819-4fe1-891e-522f0dd8b946" (UID: "29e51776-a819-4fe1-891e-522f0dd8b946"). InnerVolumeSpecName "kube-api-access-m7shm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:00.394500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.394467 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7shm\" (UniqueName: \"kubernetes.io/projected/29e51776-a819-4fe1-891e-522f0dd8b946-kube-api-access-m7shm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.394500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.394498 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e51776-a819-4fe1-891e-522f0dd8b946-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.394500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.394510 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f3acba-c988-41b2-bbd5-6fc61e1d8323-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.394743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.394519 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f3acba-c988-41b2-bbd5-6fc61e1d8323-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.394743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.394528 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhp7\" (UniqueName: \"kubernetes.io/projected/01f3acba-c988-41b2-bbd5-6fc61e1d8323-kube-api-access-7dhp7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:00.800144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.800112 5058 generic.go:356] "Generic (PLEG): container finished" podID="29e51776-a819-4fe1-891e-522f0dd8b946" containerID="9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee" exitCode=0 Oct 30 05:13:00.800627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.800172 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" Oct 30 05:13:00.800627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.800200 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" event={"ID":"29e51776-a819-4fe1-891e-522f0dd8b946","Type":"ContainerDied","Data":"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee"} Oct 30 05:13:00.800627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.800278 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28" event={"ID":"29e51776-a819-4fe1-891e-522f0dd8b946","Type":"ContainerDied","Data":"0cfeeeabc56392d6cff4890a376b1bb9391e37e116a65522bcfba06df4e2520f"} Oct 30 05:13:00.800627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.800308 5058 scope.go:122] "RemoveContainer" containerID="9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee" Oct 30 05:13:00.802332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.802311 5058 generic.go:356] "Generic (PLEG): container finished" podID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" containerID="33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a" exitCode=0 Oct 30 05:13:00.802509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.802495 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" Oct 30 05:13:00.802654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.802635 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" event={"ID":"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59","Type":"ContainerDied","Data":"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a"} Oct 30 05:13:00.802732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.802681 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw" event={"ID":"e3bfa743-8e65-47b2-90ba-bf7c4eb99f59","Type":"ContainerDied","Data":"d52272b2760c1a5989469581d094451ca5c3fbedbc45d2dffe4c2803a580ca55"} Oct 30 05:13:00.803952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.803932 5058 generic.go:356] "Generic (PLEG): container finished" podID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerID="95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958" exitCode=0 Oct 30 05:13:00.804061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.804014 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" event={"ID":"01f3acba-c988-41b2-bbd5-6fc61e1d8323","Type":"ContainerDied","Data":"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958"} Oct 30 05:13:00.804061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.804032 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" event={"ID":"01f3acba-c988-41b2-bbd5-6fc61e1d8323","Type":"ContainerDied","Data":"2c29fe6b065081907890b7c866a623617cfd19782e80a6f650ed79d918932b24"} Oct 30 05:13:00.804183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.804073 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd" Oct 30 05:13:00.817136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.817114 5058 scope.go:122] "RemoveContainer" containerID="9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee" Oct 30 05:13:00.817720 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:00.817692 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee\": container with ID starting with 9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee not found: ID does not exist" containerID="9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee" Oct 30 05:13:00.817815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.817730 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee"} err="failed to get container status \"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee\": rpc error: code = NotFound desc = could not find container \"9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee\": container with ID starting with 9bce3938054197ad60041c13d2d0f68a12b188eeb14ae059776c6664bdb387ee not found: ID does not exist" Oct 30 05:13:00.817815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.817756 5058 scope.go:122] "RemoveContainer" containerID="33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a" Oct 30 05:13:00.834878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.834853 5058 scope.go:122] "RemoveContainer" containerID="33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a" Oct 30 05:13:00.835204 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:00.835181 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a\": container with ID starting with 33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a not found: ID does not exist" containerID="33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a" Oct 30 05:13:00.835333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.835252 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a"} err="failed to get container status \"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a\": rpc error: code = NotFound desc = could not find container \"33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a\": container with ID starting with 33c2a9e725557477f128b961927eb00ecc8bf1a4e7e3745df650a2ab212a3a6a not found: ID does not exist" Oct 30 05:13:00.835333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.835278 5058 scope.go:122] "RemoveContainer" containerID="95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958" Oct 30 05:13:00.851608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.851478 5058 scope.go:122] "RemoveContainer" containerID="95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958" Oct 30 05:13:00.852127 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:00.852102 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958\": container with ID starting with 95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958 not found: ID does not exist" containerID="95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958" Oct 30 05:13:00.852251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.852133 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958"} err="failed to get container status \"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958\": rpc error: code = NotFound desc = could not find container \"95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958\": container with ID starting with 95294e124b98e3d2d5129e886c7752b53bf939efa262ac969f3eaafa60362958 not found: ID does not exist" Oct 30 05:13:00.892816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.892782 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:13:00.905930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.905897 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28"] Oct 30 05:13:00.935409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.935375 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:13:00.941795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.941765 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ac96f3/test-model-ac96f3-kserve-8476d84447-hlltd"] Oct 30 05:13:00.973458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.973378 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:13:00.982535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:00.982500 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw"] Oct 30 05:13:01.168301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:01.168261 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" path="/var/lib/kubelet/pods/01f3acba-c988-41b2-bbd5-6fc61e1d8323/volumes" Oct 30 05:13:01.169027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:01.169006 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e51776-a819-4fe1-891e-522f0dd8b946" path="/var/lib/kubelet/pods/29e51776-a819-4fe1-891e-522f0dd8b946/volumes" Oct 30 05:13:01.169670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:01.169650 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" path="/var/lib/kubelet/pods/e3bfa743-8e65-47b2-90ba-bf7c4eb99f59/volumes" Oct 30 05:13:04.844594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.844558 5058 generic.go:356] "Generic (PLEG): container finished" podID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerID="bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8" exitCode=0 Oct 30 05:13:04.845046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.844627 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerDied","Data":"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8"} Oct 30 05:13:04.845115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.845095 5058 scope.go:122] "RemoveContainer" containerID="bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8" Oct 30 05:13:04.847022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.846993 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:04.847115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.847060 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:04.847115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:04.847094 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:05.854716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:05.854673 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerStarted","Data":"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9"} Oct 30 05:13:06.131979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:06.130604 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:13:06.131979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:06.130742 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:06.131979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:06.130783 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:07.150372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:07.147847 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:07.150372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:07.147924 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:08.135958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:08.134945 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:08.135958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:08.135015 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:08.823801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:08.823769 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:13:09.154748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:09.154646 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:09.154748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:09.154717 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:10.140249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:10.130660 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:10.140249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:10.130736 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:11.158242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:11.157924 5058 patch_prober.go:29] interesting pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:13:11.158242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:11.157986 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 05:13:12.134611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:12.134582 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:13:12.135106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:12.135066 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:13:12.135759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:12.135738 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:13:17.203478 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.203442 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:13:17.204268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204251 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7808cbe3-5187-4e5f-9a32-d233eff308fb" containerName="payload-pre-processing" Oct 30 05:13:17.204329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204270 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7808cbe3-5187-4e5f-9a32-d233eff308fb" containerName="payload-pre-processing" Oct 30 05:13:17.204329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204292 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" containerName="payload-processing" Oct 30 05:13:17.204329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204298 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" containerName="payload-processing" Oct 30 05:13:17.204329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204319 5058 cpu_manager.go:414] "containerMap: removing container" podUID="29e51776-a819-4fe1-891e-522f0dd8b946" containerName="payload-pre-processing" Oct 30 05:13:17.204329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204324 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e51776-a819-4fe1-891e-522f0dd8b946" containerName="payload-pre-processing" Oct 30 05:13:17.204487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204335 5058 cpu_manager.go:414] "containerMap: removing container" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" Oct 30 05:13:17.204487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204339 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" Oct 30 05:13:17.204487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204484 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7808cbe3-5187-4e5f-9a32-d233eff308fb" containerName="payload-pre-processing" Oct 30 05:13:17.204580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204493 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bfa743-8e65-47b2-90ba-bf7c4eb99f59" containerName="payload-processing" Oct 30 05:13:17.204580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204501 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e51776-a819-4fe1-891e-522f0dd8b946" containerName="payload-pre-processing" Oct 30 05:13:17.204580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.204508 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f3acba-c988-41b2-bbd5-6fc61e1d8323" containerName="main" Oct 30 05:13:17.209557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.209537 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.248423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.248393 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-010013-serving-cert\"" Oct 30 05:13:17.253655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.253610 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.254184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.254159 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.254437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.254411 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.254565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.254550 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc87h\" (UniqueName: \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.256554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.256526 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:13:17.356241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.356196 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc87h\" (UniqueName: \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.356450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.356322 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.356574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.356559 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.356642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.356602 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.356759 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:17.356743 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-serving-cert: secret "maas-api-e2e-inf-a-010013-serving-cert" not found Oct 30 05:13:17.356827 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:17.356811 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls podName:d91a2f1f-38a4-4d93-b061-86fb9be22e38 nodeName:}" failed. No retries permitted until 2026-10-30 05:13:17.856792961 +0000 UTC m=+1770.826287946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls") pod "maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" (UID: "d91a2f1f-38a4-4d93-b061-86fb9be22e38") : secret "maas-api-e2e-inf-a-010013-serving-cert" not found Oct 30 05:13:17.357212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.357190 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.358998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.358975 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.383034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.383002 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc87h\" (UniqueName: \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.867776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.867739 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:17.870275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:17.870241 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") pod \"maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:18.124003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:18.123920 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:18.427006 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:13:18.426977 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91a2f1f_38a4_4d93_b061_86fb9be22e38.slice/crio-265d97d1bc3d7aeb036d5739018387542239943b9d2c13febec916a2a333ed6d WatchSource:0}: Error finding container 265d97d1bc3d7aeb036d5739018387542239943b9d2c13febec916a2a333ed6d: Status 404 returned error can't find the container with id 265d97d1bc3d7aeb036d5739018387542239943b9d2c13febec916a2a333ed6d Oct 30 05:13:18.434421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:18.434396 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:13:19.167502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:19.167404 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" event={"ID":"d91a2f1f-38a4-4d93-b061-86fb9be22e38","Type":"ContainerStarted","Data":"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69"} Oct 30 05:13:19.167502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:19.167478 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:19.167705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:19.167506 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" event={"ID":"d91a2f1f-38a4-4d93-b061-86fb9be22e38","Type":"ContainerStarted","Data":"265d97d1bc3d7aeb036d5739018387542239943b9d2c13febec916a2a333ed6d"} Oct 30 05:13:19.197792 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:19.197755 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:13:19.215065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:19.215011 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" podStartSLOduration=1.740422996 podStartE2EDuration="2.21499879s" totalImagesPullingTime="474.575794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:19.212043519 +0000 UTC m=+1772.181538509" watchObservedRunningTime="2026-10-30 05:13:19.21499879 +0000 UTC m=+1772.184493779" Oct 30 05:13:24.159304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.159264 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" containerID="cri-o://793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1" gracePeriod=45 Oct 30 05:13:24.475277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.475246 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:13:24.621545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.621508 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 05:13:24.621746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.621720 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" containerID="cri-o://eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8" gracePeriod=30 Oct 30 05:13:24.657557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.657518 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\" (UniqueName: \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.657777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.657634 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.657854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.657812 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.658246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.658046 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.658413 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.658432 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.658492 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\") pod \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\" (UID: \"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2\") " Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.658602 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.659202 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:24.659436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.659244 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:24.659875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.659849 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:24.661158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.661125 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:24.661390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.661358 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:24.662159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.662134 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw" pod "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" (UID: "e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2"). InnerVolumeSpecName "kube-api-access-z4sjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:24.760417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.760378 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:24.760417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.760408 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:24.760417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.760419 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4sjw\" (UniqueName: \"kubernetes.io/projected/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-kube-api-access-z4sjw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:24.760703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:24.760431 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:25.218607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.218575 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:13:25.224033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.224007 5058 generic.go:356] "Generic (PLEG): container finished" podID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerID="793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1" exitCode=0 Oct 30 05:13:25.224175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.224041 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" Oct 30 05:13:25.224175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.224081 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" event={"ID":"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2","Type":"ContainerDied","Data":"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1"} Oct 30 05:13:25.224175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.224120 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v" event={"ID":"e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2","Type":"ContainerDied","Data":"61e3f7712eab6f3351ca3ae2bbb4f7790a07ec33b0e6f6267daa529208b36c2a"} Oct 30 05:13:25.224175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.224136 5058 scope.go:122] "RemoveContainer" containerID="793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1" Oct 30 05:13:25.238799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.238775 5058 scope.go:122] "RemoveContainer" containerID="793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1" Oct 30 05:13:25.239103 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:25.239082 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1\": container with ID starting with 793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1 not found: ID does not exist" containerID="793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1" Oct 30 05:13:25.239163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.239110 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1"} err="failed to get container status \"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1\": rpc error: code = NotFound desc = could not find container \"793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1\": container with ID starting with 793df8cd1cda93c876cb067e21ea07be6d7fabe0f219e19e28972af22b4b33b1 not found: ID does not exist" Oct 30 05:13:25.329029 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.328997 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:13:25.337625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:25.337595 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4225df/test-model-4225df-kserve-b44db4d5-kwm9v"] Oct 30 05:13:27.181241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:27.173144 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" path="/var/lib/kubelet/pods/e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2/volumes" Oct 30 05:13:28.609632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:28.609590 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:13:28.610120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:28.609671 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:13:29.946558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:29.946528 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 05:13:30.083416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083382 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083438 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083497 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083582 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083669 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083789 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083869 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.083946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083900 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:30.083946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083908 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.084052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.083813 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:30.084052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084027 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\") pod \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\" (UID: \"082518df-92fd-46d1-9d1b-c8d80dc3d4d4\") " Oct 30 05:13:30.084304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084277 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:30.084304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084295 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:30.084440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084306 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:30.084885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084861 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.084976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084891 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.084976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084908 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.084976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084921 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.084976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.084934 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.086616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.086591 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:30.086720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.086610 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:30.087258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.087190 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:30.087483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.087467 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph" pod "082518df-92fd-46d1-9d1b-c8d80dc3d4d4" (UID: "082518df-92fd-46d1-9d1b-c8d80dc3d4d4"). InnerVolumeSpecName "kube-api-access-97xph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:30.186703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.186627 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.186703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.186655 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.186703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.186665 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97xph\" (UniqueName: \"kubernetes.io/projected/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-kube-api-access-97xph\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.186703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.186674 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/082518df-92fd-46d1-9d1b-c8d80dc3d4d4-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:30.300795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.300755 5058 generic.go:356] "Generic (PLEG): container finished" podID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerID="eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8" exitCode=0 Oct 30 05:13:30.300969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.300806 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" Oct 30 05:13:30.300969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.300833 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" event={"ID":"082518df-92fd-46d1-9d1b-c8d80dc3d4d4","Type":"ContainerDied","Data":"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8"} Oct 30 05:13:30.300969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.300891 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk" event={"ID":"082518df-92fd-46d1-9d1b-c8d80dc3d4d4","Type":"ContainerDied","Data":"7a8833444d53841fb7acdc078c99a641448e3574534d4c18cd7b4d64c2f452dc"} Oct 30 05:13:30.300969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.300913 5058 scope.go:122] "RemoveContainer" containerID="eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8" Oct 30 05:13:30.313632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.313609 5058 scope.go:122] "RemoveContainer" containerID="eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8" Oct 30 05:13:30.313936 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:30.313915 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8\": container with ID starting with eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8 not found: ID does not exist" containerID="eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8" Oct 30 05:13:30.314143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.313947 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8"} err="failed to get container status \"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8\": rpc error: code = NotFound desc = could not find container \"eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8\": container with ID starting with eb1edafd38a34158d53d0521c0e8c62068a6bbe6e620c300c70eb881ecd243e8 not found: ID does not exist" Oct 30 05:13:30.364396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.364359 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 05:13:30.369190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:30.369159 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk"] Oct 30 05:13:31.171651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:31.171610 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" path="/var/lib/kubelet/pods/082518df-92fd-46d1-9d1b-c8d80dc3d4d4/volumes" Oct 30 05:13:34.839232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.839187 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:13:34.840580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840551 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" Oct 30 05:13:34.840706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840582 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" Oct 30 05:13:34.840706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840629 5058 cpu_manager.go:414] "containerMap: removing container" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" Oct 30 05:13:34.840706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840637 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" Oct 30 05:13:34.840876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840861 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25df3e1-3ac5-49e8-bd47-ee15dbe7f5b2" containerName="main" Oct 30 05:13:34.840930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.840885 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="082518df-92fd-46d1-9d1b-c8d80dc3d4d4" containerName="istio-proxy" Oct 30 05:13:34.847276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.847234 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:34.861802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.861770 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:13:34.995242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.991558 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-36fd3c-dockercfg-wd72j\"" Oct 30 05:13:34.995242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:34.991660 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-36fd3c\"" Oct 30 05:13:35.146765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.146028 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.146765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.146317 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpzkw\" (UniqueName: \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.257243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.252483 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzkw\" (UniqueName: \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.257243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.252750 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.257243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.254025 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.307873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.307800 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzkw\" (UniqueName: \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\") pod \"payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.465475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.465388 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:35.766458 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:13:35.766424 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2fb18e_b987_4fe5_8e93_845d5d0a50c3.slice/crio-ca94b6834ed95b68cda6717974a5160bc3922d3334beb8cc178046643502669b WatchSource:0}: Error finding container ca94b6834ed95b68cda6717974a5160bc3922d3334beb8cc178046643502669b: Status 404 returned error can't find the container with id ca94b6834ed95b68cda6717974a5160bc3922d3334beb8cc178046643502669b Oct 30 05:13:35.768358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:35.768330 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:13:36.372004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:36.371961 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" event={"ID":"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3","Type":"ContainerStarted","Data":"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716"} Oct 30 05:13:36.372475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:36.372022 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" event={"ID":"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3","Type":"ContainerStarted","Data":"ca94b6834ed95b68cda6717974a5160bc3922d3334beb8cc178046643502669b"} Oct 30 05:13:36.372475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:36.372185 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:36.415813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:36.415756 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" podStartSLOduration=2.415737532 podStartE2EDuration="2.415737532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:36.412015449 +0000 UTC m=+1789.381510463" watchObservedRunningTime="2026-10-30 05:13:36.415737532 +0000 UTC m=+1789.385232522" Oct 30 05:13:43.180308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:43.180271 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:13:43.221408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:43.221373 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:13:47.453387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:47.453354 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:13:47.545259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:47.541419 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:13:47.566134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:47.566107 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:13:48.180618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.180577 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" podUID="5585502a-2049-40be-98dd-ea02267c56e8" containerName="payload-processing" containerID="cri-o://904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8" gracePeriod=25 Oct 30 05:13:48.222312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.222264 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" podUID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" containerName="payload-pre-processing" containerID="cri-o://c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a" gracePeriod=25 Oct 30 05:13:48.583232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.583186 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:13:48.616109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.616084 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:13:48.636389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.636355 5058 generic.go:356] "Generic (PLEG): container finished" podID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" containerID="c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a" exitCode=0 Oct 30 05:13:48.636575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.636409 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" Oct 30 05:13:48.636575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.636439 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" event={"ID":"1471b201-a8bc-4b28-bd84-8bfbefbb046b","Type":"ContainerDied","Data":"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a"} Oct 30 05:13:48.636575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.636496 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7" event={"ID":"1471b201-a8bc-4b28-bd84-8bfbefbb046b","Type":"ContainerDied","Data":"acc03e68c50cd1cd977b28ec9926a65852bda7809fbb314e8dd9603fd1bc3982"} Oct 30 05:13:48.636575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.636527 5058 scope.go:122] "RemoveContainer" containerID="c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a" Oct 30 05:13:48.637897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.637872 5058 generic.go:356] "Generic (PLEG): container finished" podID="5585502a-2049-40be-98dd-ea02267c56e8" containerID="904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8" exitCode=0 Oct 30 05:13:48.637992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.637919 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" Oct 30 05:13:48.638066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.638051 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" event={"ID":"5585502a-2049-40be-98dd-ea02267c56e8","Type":"ContainerDied","Data":"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8"} Oct 30 05:13:48.638102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.638090 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp" event={"ID":"5585502a-2049-40be-98dd-ea02267c56e8","Type":"ContainerDied","Data":"70be3568d11dc060e251510f47e35996339a75723348ebcc8c64ccc363f6ca69"} Oct 30 05:13:48.649133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649108 5058 scope.go:122] "RemoveContainer" containerID="c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a" Oct 30 05:13:48.649291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649114 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\" (UniqueName: \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\") pod \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " Oct 30 05:13:48.649363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649308 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\") pod \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\" (UID: \"1471b201-a8bc-4b28-bd84-8bfbefbb046b\") " Oct 30 05:13:48.649643 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:48.649618 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a\": container with ID starting with c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a not found: ID does not exist" containerID="c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a" Oct 30 05:13:48.649740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649651 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a"} err="failed to get container status \"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a\": rpc error: code = NotFound desc = could not find container \"c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a\": container with ID starting with c04a48761fe58209c4edd22cde4d23012c5dfc9b1a2935d9718a61072702d42a not found: ID does not exist" Oct 30 05:13:48.649740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649668 5058 scope.go:122] "RemoveContainer" containerID="904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8" Oct 30 05:13:48.650009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.649968 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume" pod "1471b201-a8bc-4b28-bd84-8bfbefbb046b" (UID: "1471b201-a8bc-4b28-bd84-8bfbefbb046b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.650144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.650123 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1471b201-a8bc-4b28-bd84-8bfbefbb046b-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.651980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.651946 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd" pod "1471b201-a8bc-4b28-bd84-8bfbefbb046b" (UID: "1471b201-a8bc-4b28-bd84-8bfbefbb046b"). InnerVolumeSpecName "kube-api-access-hxfwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.661060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.661035 5058 scope.go:122] "RemoveContainer" containerID="904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8" Oct 30 05:13:48.661472 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:48.661450 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8\": container with ID starting with 904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8 not found: ID does not exist" containerID="904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8" Oct 30 05:13:48.661556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.661487 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8"} err="failed to get container status \"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8\": rpc error: code = NotFound desc = could not find container \"904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8\": container with ID starting with 904828b01b4545e2efda5ef8aa1473300ed852de0236d9eb80f9fe07bd1813c8 not found: ID does not exist" Oct 30 05:13:48.751388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.751304 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\") pod \"5585502a-2049-40be-98dd-ea02267c56e8\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " Oct 30 05:13:48.751536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.751518 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\" (UniqueName: \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\") pod \"5585502a-2049-40be-98dd-ea02267c56e8\" (UID: \"5585502a-2049-40be-98dd-ea02267c56e8\") " Oct 30 05:13:48.751686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.751665 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume" pod "5585502a-2049-40be-98dd-ea02267c56e8" (UID: "5585502a-2049-40be-98dd-ea02267c56e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.751988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.751968 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5585502a-2049-40be-98dd-ea02267c56e8-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.751988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.751988 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfwd\" (UniqueName: \"kubernetes.io/projected/1471b201-a8bc-4b28-bd84-8bfbefbb046b-kube-api-access-hxfwd\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.753502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.753477 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52" pod "5585502a-2049-40be-98dd-ea02267c56e8" (UID: "5585502a-2049-40be-98dd-ea02267c56e8"). InnerVolumeSpecName "kube-api-access-q2r52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.854016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.853984 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2r52\" (UniqueName: \"kubernetes.io/projected/5585502a-2049-40be-98dd-ea02267c56e8-kube-api-access-q2r52\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.988695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:48.988659 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:13:49.001690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.001619 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7"] Oct 30 05:13:49.026881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.026852 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:13:49.030504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.030483 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp"] Oct 30 05:13:49.163251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.163180 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" path="/var/lib/kubelet/pods/1471b201-a8bc-4b28-bd84-8bfbefbb046b/volumes" Oct 30 05:13:49.163746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.163723 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5585502a-2049-40be-98dd-ea02267c56e8" path="/var/lib/kubelet/pods/5585502a-2049-40be-98dd-ea02267c56e8/volumes" Oct 30 05:13:49.195180 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:49.195145 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:13:49.678239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.674364 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:13:49.678239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:49.675423 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" podUID="654c6636-48fe-4a83-b0c8-344e43e2b069" containerName="maas-api" containerID="cri-o://32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22" gracePeriod=30 Oct 30 05:13:50.315846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.315818 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:13:50.395969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.395928 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\") pod \"654c6636-48fe-4a83-b0c8-344e43e2b069\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " Oct 30 05:13:50.396214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.396193 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\") pod \"654c6636-48fe-4a83-b0c8-344e43e2b069\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " Oct 30 05:13:50.396323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.396305 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\" (UniqueName: \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\") pod \"654c6636-48fe-4a83-b0c8-344e43e2b069\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " Oct 30 05:13:50.396387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.396345 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca" pod "654c6636-48fe-4a83-b0c8-344e43e2b069" (UID: "654c6636-48fe-4a83-b0c8-344e43e2b069"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:50.396448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.396420 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\") pod \"654c6636-48fe-4a83-b0c8-344e43e2b069\" (UID: \"654c6636-48fe-4a83-b0c8-344e43e2b069\") " Oct 30 05:13:50.397087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.397065 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c6636-48fe-4a83-b0c8-344e43e2b069-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:50.398656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.398635 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls" pod "654c6636-48fe-4a83-b0c8-344e43e2b069" (UID: "654c6636-48fe-4a83-b0c8-344e43e2b069"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:50.398867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.398847 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs" pod "654c6636-48fe-4a83-b0c8-344e43e2b069" (UID: "654c6636-48fe-4a83-b0c8-344e43e2b069"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:50.399046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.399023 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc" pod "654c6636-48fe-4a83-b0c8-344e43e2b069" (UID: "654c6636-48fe-4a83-b0c8-344e43e2b069"). InnerVolumeSpecName "kube-api-access-gqncc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:50.498709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.498624 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:50.498709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.498665 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqncc\" (UniqueName: \"kubernetes.io/projected/654c6636-48fe-4a83-b0c8-344e43e2b069-kube-api-access-gqncc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:50.498709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.498676 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c6636-48fe-4a83-b0c8-344e43e2b069-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:13:50.656513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.656471 5058 generic.go:356] "Generic (PLEG): container finished" podID="654c6636-48fe-4a83-b0c8-344e43e2b069" containerID="32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22" exitCode=0 Oct 30 05:13:50.656679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.656515 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" Oct 30 05:13:50.656679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.656546 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" event={"ID":"654c6636-48fe-4a83-b0c8-344e43e2b069","Type":"ContainerDied","Data":"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22"} Oct 30 05:13:50.656679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.656605 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx" event={"ID":"654c6636-48fe-4a83-b0c8-344e43e2b069","Type":"ContainerDied","Data":"8e3d23ee3c852fc34f798a39d2415e6c548170b1d589ad87a33c2073664a9a59"} Oct 30 05:13:50.656679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.656631 5058 scope.go:122] "RemoveContainer" containerID="32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22" Oct 30 05:13:50.668387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.668370 5058 scope.go:122] "RemoveContainer" containerID="32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22" Oct 30 05:13:50.668691 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:50.668672 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22\": container with ID starting with 32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22 not found: ID does not exist" containerID="32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22" Oct 30 05:13:50.668736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.668700 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22"} err="failed to get container status \"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22\": rpc error: code = NotFound desc = could not find container \"32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22\": container with ID starting with 32dd8e9d93ae29f3ca510e4a15538d62c8dcd6b5b0326923fcee809538a2ef22 not found: ID does not exist" Oct 30 05:13:50.722677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.722641 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:13:50.728634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:50.728605 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d55bd-644fb68464-gmqgx"] Oct 30 05:13:51.163502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.163469 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654c6636-48fe-4a83-b0c8-344e43e2b069" path="/var/lib/kubelet/pods/654c6636-48fe-4a83-b0c8-344e43e2b069/volumes" Oct 30 05:13:51.603310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.603231 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:13:51.604099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604077 5058 cpu_manager.go:414] "containerMap: removing container" podUID="654c6636-48fe-4a83-b0c8-344e43e2b069" containerName="maas-api" Oct 30 05:13:51.604099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604100 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654c6636-48fe-4a83-b0c8-344e43e2b069" containerName="maas-api" Oct 30 05:13:51.604305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604145 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5585502a-2049-40be-98dd-ea02267c56e8" containerName="payload-processing" Oct 30 05:13:51.604305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604153 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5585502a-2049-40be-98dd-ea02267c56e8" containerName="payload-processing" Oct 30 05:13:51.604305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604167 5058 cpu_manager.go:414] "containerMap: removing container" podUID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" containerName="payload-pre-processing" Oct 30 05:13:51.604305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604175 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" containerName="payload-pre-processing" Oct 30 05:13:51.604505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604339 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="654c6636-48fe-4a83-b0c8-344e43e2b069" containerName="maas-api" Oct 30 05:13:51.604505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604359 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5585502a-2049-40be-98dd-ea02267c56e8" containerName="payload-processing" Oct 30 05:13:51.604505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.604369 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="1471b201-a8bc-4b28-bd84-8bfbefbb046b" containerName="payload-pre-processing" Oct 30 05:13:51.609661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.609640 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.640499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.640465 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f8eb4-openshift-default-dockercfg-kktfc\"" Oct 30 05:13:51.647000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.646969 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:13:51.715099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715064 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715212 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715276 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715296 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715362 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715499 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715519 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.715684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.715667 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqxl\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.816999 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817039 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817058 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817089 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817292 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817324 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817351 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817456 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqxl\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817465 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.817607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817601 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.818032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817651 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.818032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817760 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.818032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.817834 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.818324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.818302 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.819691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.819674 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.819792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.819776 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.837154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.837134 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.837533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.837511 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqxl\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:51.924813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:51.924782 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:52.242774 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:13:52.242741 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b69f1ca_de52_471b_a19d_8be6f8914d6a.slice/crio-72bfde4dc9b9d95ef63b49105f21de7987a2f108552d454c78c87c8178cf0271 WatchSource:0}: Error finding container 72bfde4dc9b9d95ef63b49105f21de7987a2f108552d454c78c87c8178cf0271: Status 404 returned error can't find the container with id 72bfde4dc9b9d95ef63b49105f21de7987a2f108552d454c78c87c8178cf0271 Oct 30 05:13:52.244925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.244903 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:13:52.245594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.245565 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.245636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.245625 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.245666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.245655 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.681551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.681516 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" event={"ID":"2b69f1ca-de52-471b-a19d-8be6f8914d6a","Type":"ContainerStarted","Data":"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9"} Oct 30 05:13:52.681821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.681789 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" event={"ID":"2b69f1ca-de52-471b-a19d-8be6f8914d6a","Type":"ContainerStarted","Data":"72bfde4dc9b9d95ef63b49105f21de7987a2f108552d454c78c87c8178cf0271"} Oct 30 05:13:52.736077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.736015 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podStartSLOduration=1.735996782 podStartE2EDuration="1.735996782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:52.734890415 +0000 UTC m=+1805.704385404" watchObservedRunningTime="2026-10-30 05:13:52.735996782 +0000 UTC m=+1805.705491772" Oct 30 05:13:52.925508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:52.925470 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:53.949544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:53.949386 5058 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:13:53.949544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:53.949442 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 05:13:54.932142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:54.932102 5058 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:13:54.932522 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:54.932480 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 05:13:55.221380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.221248 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 05:13:55.221853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.221449 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" containerID="cri-o://ec36b2059ef023a510897bb34f01fbf5c905052e4b44fb82ee2ffb5485e16d1d" gracePeriod=30 Oct 30 05:13:55.269416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.269383 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:13:55.275885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.275860 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.283993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.283960 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:13:55.296039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.296011 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-453511-serving-cert\"" Oct 30 05:13:55.296913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.296888 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.297022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.296974 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.297022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.297003 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95gb9\" (UniqueName: \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.297131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.297071 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.397998 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.398370 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.398769 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.398869 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95gb9\" (UniqueName: \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.398994 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:55.399110 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-serving-cert: secret "maas-api-e2e-inf-b-453511-serving-cert" not found Oct 30 05:13:55.401240 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:13:55.399188 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls podName:d3fe6ff0-d65c-4759-bf28-0f47cb9dddef nodeName:}" failed. No retries permitted until 2026-10-30 05:13:55.899164117 +0000 UTC m=+1808.868659086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls") pod "maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" (UID: "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef") : secret "maas-api-e2e-inf-b-453511-serving-cert" not found Oct 30 05:13:55.404264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.402281 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.505465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.496202 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95gb9\" (UniqueName: \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.899591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.899543 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.902010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.901983 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") pod \"maas-api-e2e-inf-b-453511-d84f9cd79-8sl99\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.909836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.909806 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:55.962423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.962361 5058 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:13:55.962629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:55.962430 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 05:13:56.286926 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:13:56.286860 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fe6ff0_d65c_4759_bf28_0f47cb9dddef.slice/crio-b3f02b40f8c8809834e5499a6856159bc616983918d20791473721b5282e2283 WatchSource:0}: Error finding container b3f02b40f8c8809834e5499a6856159bc616983918d20791473721b5282e2283: Status 404 returned error can't find the container with id b3f02b40f8c8809834e5499a6856159bc616983918d20791473721b5282e2283 Oct 30 05:13:56.316622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:56.315889 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:13:56.803585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:56.803551 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" event={"ID":"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef","Type":"ContainerStarted","Data":"b3f02b40f8c8809834e5499a6856159bc616983918d20791473721b5282e2283"} Oct 30 05:13:56.929242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:56.928990 5058 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:13:56.929242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:56.929067 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 05:13:57.813605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:57.813568 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" event={"ID":"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef","Type":"ContainerStarted","Data":"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273"} Oct 30 05:13:57.814011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:57.813640 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:13:57.879874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:57.879820 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" podStartSLOduration=2.393763825 podStartE2EDuration="2.879804572s" totalImagesPullingTime="486.040747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:57.876150706 +0000 UTC m=+1810.845645695" watchObservedRunningTime="2026-10-30 05:13:57.879804572 +0000 UTC m=+1810.849299557" Oct 30 05:13:57.947243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:57.929441 5058 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:13:57.947243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:57.929511 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 05:13:58.932195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:58.932165 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:58.933170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:58.932491 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:58.933403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:58.933388 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:13:59.394265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:59.394230 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:13:59.394497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:13:59.394447 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" containerID="cri-o://12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db" gracePeriod=30 Oct 30 05:14:00.677831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.677801 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:14:00.679866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.679161 5058 patch_prober.go:29] interesting pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:14:00.680023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.679996 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:14:00.689491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.686138 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.717525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.717493 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ccafe0-openshift-default-dockercfg-9frfg\"" Oct 30 05:14:00.721100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.721043 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:14:00.842552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.842515 5058 generic.go:356] "Generic (PLEG): container finished" podID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerID="ec36b2059ef023a510897bb34f01fbf5c905052e4b44fb82ee2ffb5485e16d1d" exitCode=0 Oct 30 05:14:00.843678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.843647 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" event={"ID":"9ccd7794-b6dc-4417-b419-012fdbd4a42f","Type":"ContainerDied","Data":"ec36b2059ef023a510897bb34f01fbf5c905052e4b44fb82ee2ffb5485e16d1d"} Oct 30 05:14:00.865156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865119 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865242 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865283 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865476 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865522 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865570 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6ld\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865643 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865713 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.865766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.865746 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983075 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983127 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983161 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6ld\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983211 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983264 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983289 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983331 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983418 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.983448 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.985140 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.985503 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.986190 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:00.989260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:00.986682 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.011287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.000626 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.083949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.083580 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.106246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.096588 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.110418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.109673 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.159540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.128960 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6ld\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\") pod \"e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.196280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.180746 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 05:14:01.213990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.213780 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.213990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.213871 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214009 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214042 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214088 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214171 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214238 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214423 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.214592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.214467 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\") pod \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\" (UID: \"9ccd7794-b6dc-4417-b419-012fdbd4a42f\") " Oct 30 05:14:01.218271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.216669 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:01.218271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.216941 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:01.218271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.217108 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:01.218271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.217432 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:01.218271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.218049 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:01.219289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.219213 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:01.219704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.219678 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "kube-api-access-tmztb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:01.219836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.219816 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:01.221111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.220446 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy" pod "9ccd7794-b6dc-4417-b419-012fdbd4a42f" (UID: "9ccd7794-b6dc-4417-b419-012fdbd4a42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:01.318446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318411 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318446 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318460 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318473 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmztb\" (UniqueName: \"kubernetes.io/projected/9ccd7794-b6dc-4417-b419-012fdbd4a42f-kube-api-access-tmztb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318488 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318501 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318515 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318528 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.318703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.318541 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccd7794-b6dc-4417-b419-012fdbd4a42f-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:01.325928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.325898 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:01.674470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.674435 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:14:01.675608 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:14:01.675573 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e40f74e_f1d7_40e4_867d_5c5b6d7de9b2.slice/crio-5d7041948001c4751d8f8c6f764eaca5c6de33ad5ccf0baa7586e7711ab51af3 WatchSource:0}: Error finding container 5d7041948001c4751d8f8c6f764eaca5c6de33ad5ccf0baa7586e7711ab51af3: Status 404 returned error can't find the container with id 5d7041948001c4751d8f8c6f764eaca5c6de33ad5ccf0baa7586e7711ab51af3 Oct 30 05:14:01.679804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.679748 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:01.680654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.680375 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:01.681140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.681094 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:01.855665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.855626 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerStarted","Data":"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b"} Oct 30 05:14:01.855847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.855720 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerStarted","Data":"5d7041948001c4751d8f8c6f764eaca5c6de33ad5ccf0baa7586e7711ab51af3"} Oct 30 05:14:01.857392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.857363 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" Oct 30 05:14:01.857392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.857380 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf" event={"ID":"9ccd7794-b6dc-4417-b419-012fdbd4a42f","Type":"ContainerDied","Data":"3d185d213e6f26c12ca7b6db57a16cb8a87502dedd96bcdffe6054c84dcff5ea"} Oct 30 05:14:01.857568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.857447 5058 scope.go:122] "RemoveContainer" containerID="ec36b2059ef023a510897bb34f01fbf5c905052e4b44fb82ee2ffb5485e16d1d" Oct 30 05:14:01.903535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.903478 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podStartSLOduration=1.903462192 podStartE2EDuration="1.903462192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:01.899036131 +0000 UTC m=+1814.868531128" watchObservedRunningTime="2026-10-30 05:14:01.903462192 +0000 UTC m=+1814.872957180" Oct 30 05:14:01.945798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.945709 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 05:14:01.951856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:01.951828 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf"] Oct 30 05:14:02.327213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:02.327115 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:02.334133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:02.334099 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:02.867417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:02.867385 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:02.868339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:02.868320 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:14:03.162742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:03.162708 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" path="/var/lib/kubelet/pods/9ccd7794-b6dc-4417-b419-012fdbd4a42f/volumes" Oct 30 05:14:03.829404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:03.829370 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:14:04.770399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.770365 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:14:04.889913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.889878 5058 generic.go:356] "Generic (PLEG): container finished" podID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerID="12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db" exitCode=0 Oct 30 05:14:04.890110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.889952 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" Oct 30 05:14:04.890110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.889963 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerDied","Data":"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db"} Oct 30 05:14:04.890110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.890024 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz" event={"ID":"90d2633a-d368-4dbb-8e6d-e8808a45fcad","Type":"ContainerDied","Data":"f94b9ee9f73cc287e58feafa0303547484a43fe6d731d16d3c04fd091b0b2751"} Oct 30 05:14:04.890110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.890057 5058 scope.go:122] "RemoveContainer" containerID="12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db" Oct 30 05:14:04.902210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.902189 5058 scope.go:122] "RemoveContainer" containerID="926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5" Oct 30 05:14:04.912609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.912590 5058 scope.go:122] "RemoveContainer" containerID="12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db" Oct 30 05:14:04.912921 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:04.912903 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db\": container with ID starting with 12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db not found: ID does not exist" containerID="12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db" Oct 30 05:14:04.912978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.912930 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db"} err="failed to get container status \"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db\": rpc error: code = NotFound desc = could not find container \"12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db\": container with ID starting with 12622193d6dd5aa907e687215d0805b09e13f973f4d8e39f3cc36a134293f1db not found: ID does not exist" Oct 30 05:14:04.912978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.912948 5058 scope.go:122] "RemoveContainer" containerID="926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5" Oct 30 05:14:04.913264 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:04.913205 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5\": container with ID starting with 926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5 not found: ID does not exist" containerID="926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5" Oct 30 05:14:04.913264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.913240 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5"} err="failed to get container status \"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5\": rpc error: code = NotFound desc = could not find container \"926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5\": container with ID starting with 926243a1dc0abe88dd32c4d792654f6c752c35781c157743c7c5be9da8e9d6e5 not found: ID does not exist" Oct 30 05:14:04.916747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916729 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.916814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916807 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.916863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916851 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.916921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916910 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.916973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916943 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.917024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.916988 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.917077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917013 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:04.917077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917036 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.917199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917088 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:04.917199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917106 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.917364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917212 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\") pod \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\" (UID: \"90d2633a-d368-4dbb-8e6d-e8808a45fcad\") " Oct 30 05:14:04.917458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917372 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:04.917577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917532 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:04.917577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.917545 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:04.918085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.918061 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:04.918187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.918090 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:04.918187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.918107 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:04.918187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.918120 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:04.918187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.918150 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:04.919494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.919472 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "kube-api-access-h52r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:04.919710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.919689 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:04.919863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.919835 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:04.921021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:04.921004 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy" pod "90d2633a-d368-4dbb-8e6d-e8808a45fcad" (UID: "90d2633a-d368-4dbb-8e6d-e8808a45fcad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:05.022155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.021524 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:05.022155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.021562 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52r2\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-kube-api-access-h52r2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:05.022155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.021583 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:05.022155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.021598 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d2633a-d368-4dbb-8e6d-e8808a45fcad-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:05.234315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.234272 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:14:05.237924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:05.237898 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz"] Oct 30 05:14:06.594165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.594121 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:14:06.595166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595142 5058 cpu_manager.go:414] "containerMap: removing container" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595168 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595183 5058 cpu_manager.go:414] "containerMap: removing container" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595190 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595249 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" Oct 30 05:14:06.595297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595257 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" Oct 30 05:14:06.595543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595444 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595460 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" containerName="istio-proxy" Oct 30 05:14:06.595543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.595476 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccd7794-b6dc-4417-b419-012fdbd4a42f" containerName="istio-proxy" Oct 30 05:14:06.600900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.600878 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.619039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.619002 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:14:06.619589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.619558 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ccafe0\"" Oct 30 05:14:06.627013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.626979 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ccafe0-dockercfg-929vm\"" Oct 30 05:14:06.671242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.668756 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.671242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.668946 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4kl\" (UniqueName: \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.816241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.811930 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.816241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.812101 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4kl\" (UniqueName: \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.816241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.813569 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.833275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.829989 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:14:06.838384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.838357 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:14:06.839279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.839259 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:06.848309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.841407 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4kl\" (UniqueName: \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\") pod \"payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:06.966348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.966320 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:07.015242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.981361 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbvq\" (UniqueName: \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.015242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:06.992367 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.128261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.118632 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbvq\" (UniqueName: \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.128261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.118689 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.128261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.119930 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.141677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.141627 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbvq\" (UniqueName: \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\") pod \"payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.167114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.167082 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d2633a-d368-4dbb-8e6d-e8808a45fcad" path="/var/lib/kubelet/pods/90d2633a-d368-4dbb-8e6d-e8808a45fcad/volumes" Oct 30 05:14:07.305002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.304974 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:07.338784 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:14:07.338747 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73cd968_2ac9_4d7f_8d51_5de33781e059.slice/crio-355bc9ea87f223bc6beb489ad0309efc30ca2a1bb13fd120308ffbf4a6b47b9b WatchSource:0}: Error finding container 355bc9ea87f223bc6beb489ad0309efc30ca2a1bb13fd120308ffbf4a6b47b9b: Status 404 returned error can't find the container with id 355bc9ea87f223bc6beb489ad0309efc30ca2a1bb13fd120308ffbf4a6b47b9b Oct 30 05:14:07.340061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.340009 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:14:07.616848 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:14:07.616811 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4047510_acbb_41cf_aa18_3817bd67037c.slice/crio-f564fbbcaf1a3080fed8efe1be70fd23c5cdc4ad4c2ab5f1885f5b678fe76785 WatchSource:0}: Error finding container f564fbbcaf1a3080fed8efe1be70fd23c5cdc4ad4c2ab5f1885f5b678fe76785: Status 404 returned error can't find the container with id f564fbbcaf1a3080fed8efe1be70fd23c5cdc4ad4c2ab5f1885f5b678fe76785 Oct 30 05:14:07.617825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:07.617801 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:14:08.040295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.040255 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" event={"ID":"d4047510-acbb-41cf-aa18-3817bd67037c","Type":"ContainerStarted","Data":"bb40b0bedfac54a894442ad67509cd0ddf9f35d0373940fbfb9e2fc11120b2e8"} Oct 30 05:14:08.040482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.040322 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" event={"ID":"d4047510-acbb-41cf-aa18-3817bd67037c","Type":"ContainerStarted","Data":"f564fbbcaf1a3080fed8efe1be70fd23c5cdc4ad4c2ab5f1885f5b678fe76785"} Oct 30 05:14:08.040482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.040366 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:08.041958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.041929 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" event={"ID":"e73cd968-2ac9-4d7f-8d51-5de33781e059","Type":"ContainerStarted","Data":"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6"} Oct 30 05:14:08.042057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.041974 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" event={"ID":"e73cd968-2ac9-4d7f-8d51-5de33781e059","Type":"ContainerStarted","Data":"355bc9ea87f223bc6beb489ad0309efc30ca2a1bb13fd120308ffbf4a6b47b9b"} Oct 30 05:14:08.042057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.042010 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:08.088444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:08.087319 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" podStartSLOduration=2.087299509 podStartE2EDuration="2.087299509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:08.083186554 +0000 UTC m=+1821.052681555" watchObservedRunningTime="2026-10-30 05:14:08.087299509 +0000 UTC m=+1821.056794499" Oct 30 05:14:10.687360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:10.687309 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" podStartSLOduration=4.687293781 podStartE2EDuration="4.687293781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:08.130036024 +0000 UTC m=+1821.099531013" watchObservedRunningTime="2026-10-30 05:14:10.687293781 +0000 UTC m=+1823.656788770" Oct 30 05:14:10.691356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:10.691107 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:14:10.691356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:10.691330 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" podUID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" containerName="maas-api" containerID="cri-o://e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc" gracePeriod=30 Oct 30 05:14:11.274241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.269356 5058 generic.go:356] "Generic (PLEG): container finished" podID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" containerID="e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc" exitCode=0 Oct 30 05:14:11.274241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.273088 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:14:11.300244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.296239 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" event={"ID":"4e82fbed-b079-4c18-aa4f-47c4f8f842ca","Type":"ContainerDied","Data":"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc"} Oct 30 05:14:11.300244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.296311 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" event={"ID":"4e82fbed-b079-4c18-aa4f-47c4f8f842ca","Type":"ContainerDied","Data":"b96cbdb5709d0236445ab38b808973723a880863bc489c31816578a158965782"} Oct 30 05:14:11.300244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.296344 5058 scope.go:122] "RemoveContainer" containerID="e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc" Oct 30 05:14:11.384797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.384764 5058 scope.go:122] "RemoveContainer" containerID="e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc" Oct 30 05:14:11.386845 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:11.386749 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc\": container with ID starting with e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc not found: ID does not exist" containerID="e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc" Oct 30 05:14:11.386845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.386810 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc"} err="failed to get container status \"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc\": rpc error: code = NotFound desc = could not find container \"e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc\": container with ID starting with e55d38699ddf05a8452657b567e81b4aeaf57898a1ff9887d1fe49990873dccc not found: ID does not exist" Oct 30 05:14:11.416059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.416022 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\") pod \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " Oct 30 05:14:11.416254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.416160 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\") pod \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " Oct 30 05:14:11.416254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.416238 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\") pod \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " Oct 30 05:14:11.416432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.416416 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\" (UniqueName: \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\") pod \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\" (UID: \"4e82fbed-b079-4c18-aa4f-47c4f8f842ca\") " Oct 30 05:14:11.417678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.417457 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca" pod "4e82fbed-b079-4c18-aa4f-47c4f8f842ca" (UID: "4e82fbed-b079-4c18-aa4f-47c4f8f842ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:11.422709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.422687 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7" pod "4e82fbed-b079-4c18-aa4f-47c4f8f842ca" (UID: "4e82fbed-b079-4c18-aa4f-47c4f8f842ca"). InnerVolumeSpecName "kube-api-access-jscv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:11.422819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.422701 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls" pod "4e82fbed-b079-4c18-aa4f-47c4f8f842ca" (UID: "4e82fbed-b079-4c18-aa4f-47c4f8f842ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:11.423600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.423572 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs" pod "4e82fbed-b079-4c18-aa4f-47c4f8f842ca" (UID: "4e82fbed-b079-4c18-aa4f-47c4f8f842ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:11.517973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.517935 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:11.517973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.517968 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:11.518183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.517984 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jscv7\" (UniqueName: \"kubernetes.io/projected/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-kube-api-access-jscv7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:11.518183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:11.517997 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e82fbed-b079-4c18-aa4f-47c4f8f842ca-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:12.277034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:12.277008 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd" Oct 30 05:14:12.335646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:12.335615 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:14:12.348693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:12.348663 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47af37-565b499dff-nzspd"] Oct 30 05:14:13.163037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:13.162993 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" path="/var/lib/kubelet/pods/4e82fbed-b079-4c18-aa4f-47c4f8f842ca/volumes" Oct 30 05:14:15.366630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.366567 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:14:15.367703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.367676 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" containerName="maas-api" Oct 30 05:14:15.367703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.367702 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" containerName="maas-api" Oct 30 05:14:15.367894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.367879 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e82fbed-b079-4c18-aa4f-47c4f8f842ca" containerName="maas-api" Oct 30 05:14:15.373195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.373178 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.388322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.388294 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:14:15.396972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.396944 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-36fd3c-serving-cert\"" Oct 30 05:14:15.496668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.496637 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.496838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.496754 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.496838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.496811 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.496927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.496838 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5xj\" (UniqueName: \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.601278 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.601361 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5xj\" (UniqueName: \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.601633 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.601738 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:15.601951 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-serving-cert: secret "maas-api-e2e-inf-b-36fd3c-serving-cert" not found Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:15.602025 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls podName:05dbdf1d-648c-4b6e-a896-acddc715558c nodeName:}" failed. No retries permitted until 2026-10-30 05:14:16.102001926 +0000 UTC m=+1829.071496916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls") pod "maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" (UID: "05dbdf1d-648c-4b6e-a896-acddc715558c") : secret "maas-api-e2e-inf-b-36fd3c-serving-cert" not found Oct 30 05:14:15.606241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.603087 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.612650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.612617 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:15.638382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:15.637040 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5xj\" (UniqueName: \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:16.116926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:16.116836 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:16.119457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:16.119425 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") pod \"maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:16.290382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:16.290342 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:16.610771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:16.610740 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:14:17.321790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:17.321699 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" event={"ID":"05dbdf1d-648c-4b6e-a896-acddc715558c","Type":"ContainerStarted","Data":"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1"} Oct 30 05:14:17.321790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:17.321761 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:17.321790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:17.321785 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" event={"ID":"05dbdf1d-648c-4b6e-a896-acddc715558c","Type":"ContainerStarted","Data":"c37f03d14db381b84b03faf3323b87ab5a8d28686f273893027e35576f362219"} Oct 30 05:14:17.369346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:17.369294 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" podStartSLOduration=2.029279623 podStartE2EDuration="2.369279683s" totalImagesPullingTime="340.00006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:17.366024431 +0000 UTC m=+1830.335519421" watchObservedRunningTime="2026-10-30 05:14:17.369279683 +0000 UTC m=+1830.338774672" Oct 30 05:14:19.052631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:19.052600 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:14:19.053109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:19.052691 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:14:19.198394 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:19.198358 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:14:19.680994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:19.680960 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:14:19.681324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:19.681243 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" containerID="cri-o://1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375" gracePeriod=30 Oct 30 05:14:22.228070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:22.228033 5058 patch_prober.go:29] interesting pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:14:22.228523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:22.228090 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:14:23.335474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:23.335442 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:14:25.180252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.177341 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:14:25.337532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337428 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337487 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337609 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337672 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337705 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337725 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.337784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337777 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.338050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337826 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.338050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337855 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:25.338050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.337866 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\") pod \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\" (UID: \"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a\") " Oct 30 05:14:25.338206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338086 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:25.338324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338279 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:25.338383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338354 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:25.338672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338647 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:25.338934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338915 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.339026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338939 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.339026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338953 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.339026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338970 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.339026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.338982 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.340199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.340173 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:25.340328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.340175 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:25.340392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.340341 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "kube-api-access-99zbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:25.340447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.340430 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy" pod "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" (UID: "78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:25.441053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.441018 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zbr\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-kube-api-access-99zbr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.441053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.441048 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.441053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.441058 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.441349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.441067 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:25.453440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.453157 5058 generic.go:356] "Generic (PLEG): container finished" podID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerID="1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375" exitCode=0 Oct 30 05:14:25.453440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.453207 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerDied","Data":"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375"} Oct 30 05:14:25.453440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.453258 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" Oct 30 05:14:25.453440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.453280 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4" event={"ID":"78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a","Type":"ContainerDied","Data":"756cea45945790762a4c033f3fd53741370a0253d03c2056985918048b8adff2"} Oct 30 05:14:25.453440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.453310 5058 scope.go:122] "RemoveContainer" containerID="1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375" Oct 30 05:14:25.465813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.465786 5058 scope.go:122] "RemoveContainer" containerID="1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57" Oct 30 05:14:25.477137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.477114 5058 scope.go:122] "RemoveContainer" containerID="1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375" Oct 30 05:14:25.477443 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:25.477424 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375\": container with ID starting with 1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375 not found: ID does not exist" containerID="1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375" Oct 30 05:14:25.477502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.477448 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375"} err="failed to get container status \"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375\": rpc error: code = NotFound desc = could not find container \"1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375\": container with ID starting with 1e3e0cd4fd7b3b9bc89dbf9f0bbf239e85a7c2eb92bd48f12f1c19a591a40375 not found: ID does not exist" Oct 30 05:14:25.477502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.477465 5058 scope.go:122] "RemoveContainer" containerID="1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57" Oct 30 05:14:25.477692 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:25.477676 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57\": container with ID starting with 1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57 not found: ID does not exist" containerID="1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57" Oct 30 05:14:25.477727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.477697 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57"} err="failed to get container status \"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57\": rpc error: code = NotFound desc = could not find container \"1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57\": container with ID starting with 1f73dd6f0d66e6d02ef00c4b1dace91b26ba706f73ce073044ef458ee8128d57 not found: ID does not exist" Oct 30 05:14:25.512344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.512306 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:14:25.516204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:25.516166 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4"] Oct 30 05:14:26.519426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.519394 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:14:26.520185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.520170 5058 cpu_manager.go:414] "containerMap: removing container" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.520251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.520186 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.520362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.520352 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.520408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.520368 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.525889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.525871 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.542502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.542474 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:14:26.551521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.551491 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-356cae\"" Oct 30 05:14:26.575404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.575372 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:14:26.576196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.576177 5058 cpu_manager.go:414] "containerMap: removing container" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.576196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.576194 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" containerName="istio-proxy" Oct 30 05:14:26.580306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.580284 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.599107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.599075 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:14:26.683241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.671511 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-356cae-dockercfg-h28t9\"" Oct 30 05:14:26.696244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.691317 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7xb\" (UniqueName: \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.696244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.691413 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.696244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.691488 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjt6p\" (UniqueName: \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.696244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.691542 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.801323 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7xb\" (UniqueName: \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.801433 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.801517 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt6p\" (UniqueName: \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.801577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.802649 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.806298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.804168 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.859242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.850731 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt6p\" (UniqueName: \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\") pod \"payload-processing-e2e-inf-a-356cae-795777684b-ftbfw\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.859242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.852512 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:26.882420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.871938 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7xb\" (UniqueName: \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\") pod \"payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:26.904709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:26.904271 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:27.164208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.164174 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a" path="/var/lib/kubelet/pods/78fdebf5-5fda-4d80-82f8-06b1a7bb5f2a/volumes" Oct 30 05:14:27.201020 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:14:27.200990 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72695511_cdb2_45dd_87c0_f5cdb6c70ae9.slice/crio-c91ecea437eb1120ed6c82af71e469b2d8feb231137b9ebd203dbeeb2a0dc7fb WatchSource:0}: Error finding container c91ecea437eb1120ed6c82af71e469b2d8feb231137b9ebd203dbeeb2a0dc7fb: Status 404 returned error can't find the container with id c91ecea437eb1120ed6c82af71e469b2d8feb231137b9ebd203dbeeb2a0dc7fb Oct 30 05:14:27.202399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.202374 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:14:27.274784 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:14:27.274755 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0798140_c1d7_4bb9_aaa0_f154c04e5f30.slice/crio-a866b7a33f4b111f09b2046984a546234a90294a240374679204ac41551d6018 WatchSource:0}: Error finding container a866b7a33f4b111f09b2046984a546234a90294a240374679204ac41551d6018: Status 404 returned error can't find the container with id a866b7a33f4b111f09b2046984a546234a90294a240374679204ac41551d6018 Oct 30 05:14:27.277068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.277045 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:14:27.472225 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.472191 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" event={"ID":"b0798140-c1d7-4bb9-aaa0-f154c04e5f30","Type":"ContainerStarted","Data":"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb"} Oct 30 05:14:27.472430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.472264 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:27.472430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.472287 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" event={"ID":"b0798140-c1d7-4bb9-aaa0-f154c04e5f30","Type":"ContainerStarted","Data":"a866b7a33f4b111f09b2046984a546234a90294a240374679204ac41551d6018"} Oct 30 05:14:27.473906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.473882 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" event={"ID":"72695511-cdb2-45dd-87c0-f5cdb6c70ae9","Type":"ContainerStarted","Data":"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec"} Oct 30 05:14:27.474034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.473924 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" event={"ID":"72695511-cdb2-45dd-87c0-f5cdb6c70ae9","Type":"ContainerStarted","Data":"c91ecea437eb1120ed6c82af71e469b2d8feb231137b9ebd203dbeeb2a0dc7fb"} Oct 30 05:14:27.474034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.473944 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:27.514068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.514017 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" podStartSLOduration=1.514003442 podStartE2EDuration="1.514003442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:27.511798873 +0000 UTC m=+1840.481293863" watchObservedRunningTime="2026-10-30 05:14:27.514003442 +0000 UTC m=+1840.483498430" Oct 30 05:14:27.556683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:27.556628 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" podStartSLOduration=1.556614701 podStartE2EDuration="1.556614701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:27.553541131 +0000 UTC m=+1840.523036121" watchObservedRunningTime="2026-10-30 05:14:27.556614701 +0000 UTC m=+1840.526109728" Oct 30 05:14:37.180153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.180120 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:14:37.180700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.180374 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" podUID="c6f02ae7-e23f-441d-8abf-80afa1624f66" containerName="maas-api" containerID="cri-o://18ddb24f8dbd7d614a9961628e1402e234cdb2bde4b2044e3e063f5072b5519c" gracePeriod=30 Oct 30 05:14:37.638452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.638070 5058 generic.go:356] "Generic (PLEG): container finished" podID="c6f02ae7-e23f-441d-8abf-80afa1624f66" containerID="18ddb24f8dbd7d614a9961628e1402e234cdb2bde4b2044e3e063f5072b5519c" exitCode=0 Oct 30 05:14:37.638452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.638371 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" event={"ID":"c6f02ae7-e23f-441d-8abf-80afa1624f66","Type":"ContainerDied","Data":"18ddb24f8dbd7d614a9961628e1402e234cdb2bde4b2044e3e063f5072b5519c"} Oct 30 05:14:37.694255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.687054 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:14:37.771237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.765868 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\") pod \"c6f02ae7-e23f-441d-8abf-80afa1624f66\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " Oct 30 05:14:37.771237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.765927 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\") pod \"c6f02ae7-e23f-441d-8abf-80afa1624f66\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " Oct 30 05:14:37.771237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.765982 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\" (UniqueName: \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\") pod \"c6f02ae7-e23f-441d-8abf-80afa1624f66\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " Oct 30 05:14:37.771237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.766023 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\") pod \"c6f02ae7-e23f-441d-8abf-80afa1624f66\" (UID: \"c6f02ae7-e23f-441d-8abf-80afa1624f66\") " Oct 30 05:14:37.772337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.772292 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca" pod "c6f02ae7-e23f-441d-8abf-80afa1624f66" (UID: "c6f02ae7-e23f-441d-8abf-80afa1624f66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:37.798427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.787402 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls" pod "c6f02ae7-e23f-441d-8abf-80afa1624f66" (UID: "c6f02ae7-e23f-441d-8abf-80afa1624f66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:37.798427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.789345 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs" pod "c6f02ae7-e23f-441d-8abf-80afa1624f66" (UID: "c6f02ae7-e23f-441d-8abf-80afa1624f66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:37.817275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.816954 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c" pod "c6f02ae7-e23f-441d-8abf-80afa1624f66" (UID: "c6f02ae7-e23f-441d-8abf-80afa1624f66"). InnerVolumeSpecName "kube-api-access-4tr7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:37.870460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.870283 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f02ae7-e23f-441d-8abf-80afa1624f66-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:37.870460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.870325 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:37.870460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.870341 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tr7c\" (UniqueName: \"kubernetes.io/projected/c6f02ae7-e23f-441d-8abf-80afa1624f66-kube-api-access-4tr7c\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:37.870460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:37.870355 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f02ae7-e23f-441d-8abf-80afa1624f66-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:38.485862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.485782 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:14:38.486237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.485871 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:14:38.647479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.647447 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" Oct 30 05:14:38.647479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.647463 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww" event={"ID":"c6f02ae7-e23f-441d-8abf-80afa1624f66","Type":"ContainerDied","Data":"7a994183621698b881dbac92b72e8eec585d60e8b926c4a39a4fa0d2720be39f"} Oct 30 05:14:38.647695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.647523 5058 scope.go:122] "RemoveContainer" containerID="18ddb24f8dbd7d614a9961628e1402e234cdb2bde4b2044e3e063f5072b5519c" Oct 30 05:14:38.707943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.707912 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:14:38.711982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:38.711955 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0814c-5856f8f656-6z2ww"] Oct 30 05:14:39.163350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:39.163321 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f02ae7-e23f-441d-8abf-80afa1624f66" path="/var/lib/kubelet/pods/c6f02ae7-e23f-441d-8abf-80afa1624f66/volumes" Oct 30 05:14:45.715246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.715173 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:14:45.716451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.716427 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c6f02ae7-e23f-441d-8abf-80afa1624f66" containerName="maas-api" Oct 30 05:14:45.716537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.716455 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f02ae7-e23f-441d-8abf-80afa1624f66" containerName="maas-api" Oct 30 05:14:45.716845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.716821 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f02ae7-e23f-441d-8abf-80afa1624f66" containerName="maas-api" Oct 30 05:14:45.722843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.722825 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.740877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.740843 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:14:45.746684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.746650 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ccafe0-serving-cert\"" Oct 30 05:14:45.816538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.816501 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:14:45.816765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.816739 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" containerID="cri-o://246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292" gracePeriod=30 Oct 30 05:14:45.848107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.848070 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.848344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.848244 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.848428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.848412 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.848580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.848566 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwq5\" (UniqueName: \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.949499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.949457 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwq5\" (UniqueName: \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.949695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.949546 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.949695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.949597 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.949822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.949705 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.950383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.950358 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.952104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.952084 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.952182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.952150 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:45.970806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:45.970710 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwq5\" (UniqueName: \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\") pod \"maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:46.066244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:46.054714 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:46.530758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:46.530720 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:14:46.731867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:46.729656 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" event={"ID":"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee","Type":"ContainerStarted","Data":"4443bb0774ddf22047720c6c140415c1a8c549885ec3852f198c03b169edc2f1"} Oct 30 05:14:47.739058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:47.739023 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" event={"ID":"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee","Type":"ContainerStarted","Data":"6c0e6c8a7b8abc3d07a24ff5b43529053806bec508196412de40caaee0a485d5"} Oct 30 05:14:47.739521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:47.739091 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:47.783201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:47.783146 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" podStartSLOduration=2.465145059 podStartE2EDuration="2.783130315s" totalImagesPullingTime="317.985256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:47.780696183 +0000 UTC m=+1860.750191171" watchObservedRunningTime="2026-10-30 05:14:47.783130315 +0000 UTC m=+1860.752625302" Oct 30 05:14:49.192737 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:49.192699 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:14:51.537830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.537802 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.666811 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.666868 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.666910 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.666989 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.667041 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.667115 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.667238 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.667280 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.668239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.667370 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\") pod \"01dda135-a34a-4e91-8175-5bd3c53ae67b\" (UID: \"01dda135-a34a-4e91-8175-5bd3c53ae67b\") " Oct 30 05:14:51.716264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.715648 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:51.718274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.718188 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:51.718471 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.718450 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:51.723091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.723046 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:51.723482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.723448 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:51.723482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.723461 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:51.723814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.723796 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:51.724294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.724273 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:51.749688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.749420 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5" pod "01dda135-a34a-4e91-8175-5bd3c53ae67b" (UID: "01dda135-a34a-4e91-8175-5bd3c53ae67b"). InnerVolumeSpecName "kube-api-access-5skx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777503 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777541 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777554 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777567 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777581 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5skx5\" (UniqueName: \"kubernetes.io/projected/01dda135-a34a-4e91-8175-5bd3c53ae67b-kube-api-access-5skx5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.777591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777594 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.778089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777607 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dda135-a34a-4e91-8175-5bd3c53ae67b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.778089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777621 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.778089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.777632 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dda135-a34a-4e91-8175-5bd3c53ae67b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:51.784399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.784151 5058 generic.go:356] "Generic (PLEG): container finished" podID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerID="246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292" exitCode=0 Oct 30 05:14:51.784399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.784232 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerDied","Data":"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292"} Oct 30 05:14:51.784399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.784269 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" event={"ID":"01dda135-a34a-4e91-8175-5bd3c53ae67b","Type":"ContainerDied","Data":"22f8cac94d75d1229db6cb8894c8d4796695bdcb057739b516fb7c1b34f79bae"} Oct 30 05:14:51.784399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.784291 5058 scope.go:122] "RemoveContainer" containerID="246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292" Oct 30 05:14:51.784399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.784326 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4" Oct 30 05:14:51.797494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.797465 5058 scope.go:122] "RemoveContainer" containerID="40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db" Oct 30 05:14:51.811798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.811641 5058 scope.go:122] "RemoveContainer" containerID="246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292" Oct 30 05:14:51.812040 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:51.812013 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292\": container with ID starting with 246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292 not found: ID does not exist" containerID="246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292" Oct 30 05:14:51.812103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.812052 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292"} err="failed to get container status \"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292\": rpc error: code = NotFound desc = could not find container \"246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292\": container with ID starting with 246e21bc2146ec51783e3b5b6db995e2d246e07f2ed4f86b7a448b4f836b7292 not found: ID does not exist" Oct 30 05:14:51.812103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.812076 5058 scope.go:122] "RemoveContainer" containerID="40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db" Oct 30 05:14:51.812607 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:51.812583 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db\": container with ID starting with 40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db not found: ID does not exist" containerID="40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db" Oct 30 05:14:51.812695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.812614 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db"} err="failed to get container status \"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db\": rpc error: code = NotFound desc = could not find container \"40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db\": container with ID starting with 40d06b623b57dfd533478becc5ff571da318f3c4c3c8a2e227eafcb06e4234db not found: ID does not exist" Oct 30 05:14:51.846173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.846139 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:14:51.849762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:51.849738 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4"] Oct 30 05:14:53.162674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:53.162639 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" path="/var/lib/kubelet/pods/01dda135-a34a-4e91-8175-5bd3c53ae67b/volumes" Oct 30 05:14:53.757744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:53.757714 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:14:56.788299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:56.788264 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:14:56.788728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:56.788489 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" podUID="3d050b17-db54-4946-a932-734e6b62079b" containerName="maas-api" containerID="cri-o://be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae" gracePeriod=30 Oct 30 05:14:56.910996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:56.910965 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:14:56.945835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:56.945803 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:14:57.109551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.109523 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:14:57.227498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.227463 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\") pod \"3d050b17-db54-4946-a932-734e6b62079b\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " Oct 30 05:14:57.227660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.227512 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\") pod \"3d050b17-db54-4946-a932-734e6b62079b\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " Oct 30 05:14:57.227848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.227826 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\" (UniqueName: \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\") pod \"3d050b17-db54-4946-a932-734e6b62079b\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " Oct 30 05:14:57.227945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.227929 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\") pod \"3d050b17-db54-4946-a932-734e6b62079b\" (UID: \"3d050b17-db54-4946-a932-734e6b62079b\") " Oct 30 05:14:57.228420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.228391 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca" pod "3d050b17-db54-4946-a932-734e6b62079b" (UID: "3d050b17-db54-4946-a932-734e6b62079b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:57.228901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.228878 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d050b17-db54-4946-a932-734e6b62079b-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:57.229984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.229955 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls" pod "3d050b17-db54-4946-a932-734e6b62079b" (UID: "3d050b17-db54-4946-a932-734e6b62079b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:57.230212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.230190 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5" pod "3d050b17-db54-4946-a932-734e6b62079b" (UID: "3d050b17-db54-4946-a932-734e6b62079b"). InnerVolumeSpecName "kube-api-access-767s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:57.231016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.230994 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs" pod "3d050b17-db54-4946-a932-734e6b62079b" (UID: "3d050b17-db54-4946-a932-734e6b62079b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:57.356323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.339089 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767s5\" (UniqueName: \"kubernetes.io/projected/3d050b17-db54-4946-a932-734e6b62079b-kube-api-access-767s5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:57.356323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.339143 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:57.356323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.339158 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d050b17-db54-4946-a932-734e6b62079b-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:14:57.845731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.845694 5058 generic.go:356] "Generic (PLEG): container finished" podID="3d050b17-db54-4946-a932-734e6b62079b" containerID="be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae" exitCode=0 Oct 30 05:14:57.846151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.845747 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" Oct 30 05:14:57.846151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.845751 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" event={"ID":"3d050b17-db54-4946-a932-734e6b62079b","Type":"ContainerDied","Data":"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae"} Oct 30 05:14:57.846151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.845802 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl" event={"ID":"3d050b17-db54-4946-a932-734e6b62079b","Type":"ContainerDied","Data":"eb4305a0fa230d20ab028a35975135c13d6b8a4db62194919b2f9bff495f1f50"} Oct 30 05:14:57.846151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.845835 5058 scope.go:122] "RemoveContainer" containerID="be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae" Oct 30 05:14:57.859053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.859033 5058 scope.go:122] "RemoveContainer" containerID="be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae" Oct 30 05:14:57.859336 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:14:57.859312 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae\": container with ID starting with be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae not found: ID does not exist" containerID="be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae" Oct 30 05:14:57.859397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.859345 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae"} err="failed to get container status \"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae\": rpc error: code = NotFound desc = could not find container \"be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae\": container with ID starting with be93d2d508e1f1908ba2687d9dcfec0af51f1bf821ace6be52753c2674887bae not found: ID does not exist" Oct 30 05:14:57.915347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.915300 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:14:57.929271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:57.929241 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4225df-b4467595f-ww8cl"] Oct 30 05:14:59.163202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:14:59.163172 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d050b17-db54-4946-a932-734e6b62079b" path="/var/lib/kubelet/pods/3d050b17-db54-4946-a932-734e6b62079b/volumes" Oct 30 05:15:01.911320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:01.911277 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" podUID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" containerName="payload-processing" containerID="cri-o://0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8" gracePeriod=25 Oct 30 05:15:01.947133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:01.947077 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" podUID="36b03b41-2e1b-48cf-8540-75b351ffaafb" containerName="payload-pre-processing" containerID="cri-o://701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8" gracePeriod=25 Oct 30 05:15:02.269118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.269088 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:15:02.341361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.341334 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:15:02.354299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.354265 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\" (UniqueName: \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\") pod \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " Oct 30 05:15:02.354543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.354526 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\") pod \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\" (UID: \"5195b4bc-acd4-42cc-9cda-d6d2c2458785\") " Oct 30 05:15:02.355112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.355084 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume" pod "5195b4bc-acd4-42cc-9cda-d6d2c2458785" (UID: "5195b4bc-acd4-42cc-9cda-d6d2c2458785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:02.355521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.355503 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5195b4bc-acd4-42cc-9cda-d6d2c2458785-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:15:02.357651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.357628 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6" pod "5195b4bc-acd4-42cc-9cda-d6d2c2458785" (UID: "5195b4bc-acd4-42cc-9cda-d6d2c2458785"). InnerVolumeSpecName "kube-api-access-tt8m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:02.456837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.456799 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\") pod \"36b03b41-2e1b-48cf-8540-75b351ffaafb\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " Oct 30 05:15:02.457025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.456874 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\" (UniqueName: \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\") pod \"36b03b41-2e1b-48cf-8540-75b351ffaafb\" (UID: \"36b03b41-2e1b-48cf-8540-75b351ffaafb\") " Oct 30 05:15:02.457349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.457323 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume" pod "36b03b41-2e1b-48cf-8540-75b351ffaafb" (UID: "36b03b41-2e1b-48cf-8540-75b351ffaafb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:02.457568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.457551 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8m6\" (UniqueName: \"kubernetes.io/projected/5195b4bc-acd4-42cc-9cda-d6d2c2458785-kube-api-access-tt8m6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:15:02.457630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.457573 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b03b41-2e1b-48cf-8540-75b351ffaafb-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:15:02.458998 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.458979 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv" pod "36b03b41-2e1b-48cf-8540-75b351ffaafb" (UID: "36b03b41-2e1b-48cf-8540-75b351ffaafb"). InnerVolumeSpecName "kube-api-access-smsbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:02.559244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.559137 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smsbv\" (UniqueName: \"kubernetes.io/projected/36b03b41-2e1b-48cf-8540-75b351ffaafb-kube-api-access-smsbv\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:15:02.913020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.912985 5058 generic.go:356] "Generic (PLEG): container finished" podID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" containerID="0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8" exitCode=0 Oct 30 05:15:02.913551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.913038 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" Oct 30 05:15:02.913551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.913069 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" event={"ID":"5195b4bc-acd4-42cc-9cda-d6d2c2458785","Type":"ContainerDied","Data":"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8"} Oct 30 05:15:02.913551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.913120 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc" event={"ID":"5195b4bc-acd4-42cc-9cda-d6d2c2458785","Type":"ContainerDied","Data":"d83c462bf62b9886afc9982a0794bdea0b89deff59e3d5db327d1f1333de7e78"} Oct 30 05:15:02.913551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.913144 5058 scope.go:122] "RemoveContainer" containerID="0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8" Oct 30 05:15:02.914657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.914633 5058 generic.go:356] "Generic (PLEG): container finished" podID="36b03b41-2e1b-48cf-8540-75b351ffaafb" containerID="701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8" exitCode=0 Oct 30 05:15:02.914758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.914670 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" Oct 30 05:15:02.914758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.914684 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" event={"ID":"36b03b41-2e1b-48cf-8540-75b351ffaafb","Type":"ContainerDied","Data":"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8"} Oct 30 05:15:02.914758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.914716 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk" event={"ID":"36b03b41-2e1b-48cf-8540-75b351ffaafb","Type":"ContainerDied","Data":"d0a41a802ffa154313bfeca50ecb447cf8a1c49d1635cd4fe012a338ef07117a"} Oct 30 05:15:02.924066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.924048 5058 scope.go:122] "RemoveContainer" containerID="0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8" Oct 30 05:15:02.924429 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:02.924393 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8\": container with ID starting with 0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8 not found: ID does not exist" containerID="0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8" Oct 30 05:15:02.924517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.924435 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8"} err="failed to get container status \"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8\": rpc error: code = NotFound desc = could not find container \"0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8\": container with ID starting with 0e2ad48b7d60062a83a1865011b365080a24542463b2afe7c66ef84950b674a8 not found: ID does not exist" Oct 30 05:15:02.924517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.924454 5058 scope.go:122] "RemoveContainer" containerID="701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8" Oct 30 05:15:02.942783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.942762 5058 scope.go:122] "RemoveContainer" containerID="701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8" Oct 30 05:15:02.943104 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:02.943083 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8\": container with ID starting with 701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8 not found: ID does not exist" containerID="701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8" Oct 30 05:15:02.943165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.943113 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8"} err="failed to get container status \"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8\": rpc error: code = NotFound desc = could not find container \"701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8\": container with ID starting with 701395109f1d6b0befe6fe525fd99f5750e4c66ebf8f22ae1b414b11ab808ee8 not found: ID does not exist" Oct 30 05:15:02.992776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:02.992741 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:15:03.005915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:03.005884 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk"] Oct 30 05:15:03.042984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:03.042952 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:15:03.047585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:03.047555 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc"] Oct 30 05:15:03.163138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:03.163058 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b03b41-2e1b-48cf-8540-75b351ffaafb" path="/var/lib/kubelet/pods/36b03b41-2e1b-48cf-8540-75b351ffaafb/volumes" Oct 30 05:15:03.163544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:03.163530 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" path="/var/lib/kubelet/pods/5195b4bc-acd4-42cc-9cda-d6d2c2458785/volumes" Oct 30 05:15:08.615010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.614972 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:15:08.615818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615801 5058 cpu_manager.go:414] "containerMap: removing container" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:08.615861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615820 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:08.615861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615830 5058 cpu_manager.go:414] "containerMap: removing container" podUID="36b03b41-2e1b-48cf-8540-75b351ffaafb" containerName="payload-pre-processing" Oct 30 05:15:08.615861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615835 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b03b41-2e1b-48cf-8540-75b351ffaafb" containerName="payload-pre-processing" Oct 30 05:15:08.615861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615849 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3d050b17-db54-4946-a932-734e6b62079b" containerName="maas-api" Oct 30 05:15:08.615861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615853 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d050b17-db54-4946-a932-734e6b62079b" containerName="maas-api" Oct 30 05:15:08.616015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615868 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" containerName="payload-processing" Oct 30 05:15:08.616015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615873 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" containerName="payload-processing" Oct 30 05:15:08.616015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615901 5058 cpu_manager.go:414] "containerMap: removing container" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:08.616015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.615906 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:08.616015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.616009 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:08.616182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.616016 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b03b41-2e1b-48cf-8540-75b351ffaafb" containerName="payload-pre-processing" Oct 30 05:15:08.616182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.616025 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d050b17-db54-4946-a932-734e6b62079b" containerName="maas-api" Oct 30 05:15:08.616182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.616032 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5195b4bc-acd4-42cc-9cda-d6d2c2458785" containerName="payload-processing" Oct 30 05:15:08.621527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.621505 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.650721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.650692 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.650884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.650753 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.650958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.650876 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.650958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.650944 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gpw\" (UniqueName: \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.660673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.660646 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:15:08.672254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.672208 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-356cae-serving-cert\"" Oct 30 05:15:08.752157 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.752111 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.752365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.752193 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.752365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.752260 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gpw\" (UniqueName: \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.752472 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:08.752393 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-serving-cert: secret "maas-api-e2e-inf-a-356cae-serving-cert" not found Oct 30 05:15:08.752472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.752441 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.752535 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:08.752474 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls podName:f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac nodeName:}" failed. No retries permitted until 2026-10-30 05:15:09.252452892 +0000 UTC m=+1882.221947863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls") pod "maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" (UID: "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac") : secret "maas-api-e2e-inf-a-356cae-serving-cert" not found Oct 30 05:15:08.753017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.752993 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.754830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.754803 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:08.776417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:08.776385 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gpw\" (UniqueName: \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:09.248369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.248160 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:15:09.249462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.249437 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dda135-a34a-4e91-8175-5bd3c53ae67b" containerName="istio-proxy" Oct 30 05:15:09.253446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.253422 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.263990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.263963 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264033 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:09.264273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264064 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264160 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264190 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264272 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264299 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264380 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264412 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.264599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.264543 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8z77\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.267644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.267599 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:09.280700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.277058 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2f6fed-openshift-default-dockercfg-7qh2j\"" Oct 30 05:15:09.293494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.293456 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:15:09.366285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366251 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8z77\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366313 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366354 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366429 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366457 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366520 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366546 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366626 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366858 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.366957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366939 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.367017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.366992 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.367092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.367069 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.367369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.367351 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.368830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.368812 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.368958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.368943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.395105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.395080 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.395277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.395079 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8z77\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\") pod \"e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:09.540590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.540496 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:09.653238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:09.610325 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:10.055869 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:10.055832 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b7ed6d_0bbb_421f_a02c_0c4fe6ced4ac.slice/crio-2565b97667aebed9e7948ab3025a18c0bdb3f9982d64a168594d9bd7d7f6b2dd WatchSource:0}: Error finding container 2565b97667aebed9e7948ab3025a18c0bdb3f9982d64a168594d9bd7d7f6b2dd: Status 404 returned error can't find the container with id 2565b97667aebed9e7948ab3025a18c0bdb3f9982d64a168594d9bd7d7f6b2dd Oct 30 05:15:10.057901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:10.057877 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:15:10.093410 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:10.093375 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1bbf5b_7760_4207_b379_b4ee036f8eb8.slice/crio-8205346d6f712c6e9e82c2d012c3841797718d8638c7456da69846a5d7b59498 WatchSource:0}: Error finding container 8205346d6f712c6e9e82c2d012c3841797718d8638c7456da69846a5d7b59498: Status 404 returned error can't find the container with id 8205346d6f712c6e9e82c2d012c3841797718d8638c7456da69846a5d7b59498 Oct 30 05:15:10.096232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:10.096192 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:15:10.096650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:10.096619 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:10.096695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:10.096685 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:10.096729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:10.096715 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:11.001817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.001722 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" event={"ID":"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac","Type":"ContainerStarted","Data":"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813"} Oct 30 05:15:11.001817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.001800 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" event={"ID":"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac","Type":"ContainerStarted","Data":"2565b97667aebed9e7948ab3025a18c0bdb3f9982d64a168594d9bd7d7f6b2dd"} Oct 30 05:15:11.002382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.002360 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:11.024022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.023983 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerStarted","Data":"987fd5b39a006fc3bec357e67441f619d2b512064584ddf81dce0d939bf9e39e"} Oct 30 05:15:11.024186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.024050 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerStarted","Data":"8205346d6f712c6e9e82c2d012c3841797718d8638c7456da69846a5d7b59498"} Oct 30 05:15:11.056257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.049526 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" podStartSLOduration=2.439561783 podStartE2EDuration="3.049505683s" totalImagesPullingTime="609.9439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:11.047676889 +0000 UTC m=+1884.017171879" watchObservedRunningTime="2026-10-30 05:15:11.049505683 +0000 UTC m=+1884.019000672" Oct 30 05:15:11.100910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.100859 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podStartSLOduration=2.100842418 podStartE2EDuration="2.100842418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:11.100609979 +0000 UTC m=+1884.070104973" watchObservedRunningTime="2026-10-30 05:15:11.100842418 +0000 UTC m=+1884.070337406" Oct 30 05:15:11.610592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.610559 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:11.617055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:11.617029 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:12.031145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:12.031112 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:12.032139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:12.032120 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:15:15.576639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.576602 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:15:15.581751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.581728 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.601636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.601599 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:15:15.608726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.608694 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-2f6fed\"" Oct 30 05:15:15.627518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.627483 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:15:15.632531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.632500 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.646760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.646728 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:15:15.650615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.650590 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-2f6fed-dockercfg-xvr5q\"" Oct 30 05:15:15.653514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.653477 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.653669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.653645 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.653743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.653707 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsz2\" (UniqueName: \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.653805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.653749 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpvd\" (UniqueName: \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.763574 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.763742 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.763813 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsz2\" (UniqueName: \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.763869 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpvd\" (UniqueName: \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.765167 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.778240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.765933 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.857286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.856266 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsz2\" (UniqueName: \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\") pod \"payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.914121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.884967 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpvd\" (UniqueName: \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\") pod \"payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:15.933995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.929641 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:15.956372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:15.956343 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:16.255473 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:16.255442 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7c7593_101f_4678_ad91_dad68d4bb26b.slice/crio-82a2791e3501bf193fa9883aee22381d54ef7e072206279063fad50a806bb5fb WatchSource:0}: Error finding container 82a2791e3501bf193fa9883aee22381d54ef7e072206279063fad50a806bb5fb: Status 404 returned error can't find the container with id 82a2791e3501bf193fa9883aee22381d54ef7e072206279063fad50a806bb5fb Oct 30 05:15:16.258672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:16.258650 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:15:16.290624 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:16.290596 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59448e72_ab39_48b0_b703_66ab864916b3.slice/crio-5a24cd6f115bbaf893d31f74dbf787193c72886f04f3003c41ddc532c92435bd WatchSource:0}: Error finding container 5a24cd6f115bbaf893d31f74dbf787193c72886f04f3003c41ddc532c92435bd: Status 404 returned error can't find the container with id 5a24cd6f115bbaf893d31f74dbf787193c72886f04f3003c41ddc532c92435bd Oct 30 05:15:16.292114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:16.292090 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:15:17.108242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.107085 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:15:17.224807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.215391 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:17.224807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.215454 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" event={"ID":"4c7c7593-101f-4678-ad91-dad68d4bb26b","Type":"ContainerStarted","Data":"fa472f1a95a8e46031d8d13a05957ca693636c22994b2d97ebcd5b9cd776c986"} Oct 30 05:15:17.224807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.215485 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" event={"ID":"4c7c7593-101f-4678-ad91-dad68d4bb26b","Type":"ContainerStarted","Data":"82a2791e3501bf193fa9883aee22381d54ef7e072206279063fad50a806bb5fb"} Oct 30 05:15:17.229623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.229521 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" event={"ID":"59448e72-ab39-48b0-b703-66ab864916b3","Type":"ContainerStarted","Data":"6818a61977e6849690bdb181a24321d9a974683251ab44f4c7a2fe5af5a9f8d3"} Oct 30 05:15:17.229623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.229580 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" event={"ID":"59448e72-ab39-48b0-b703-66ab864916b3","Type":"ContainerStarted","Data":"5a24cd6f115bbaf893d31f74dbf787193c72886f04f3003c41ddc532c92435bd"} Oct 30 05:15:17.229883 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.229754 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:17.294274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.294192 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" podStartSLOduration=2.294172851 podStartE2EDuration="2.294172851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:17.291799437 +0000 UTC m=+1890.261294426" watchObservedRunningTime="2026-10-30 05:15:17.294172851 +0000 UTC m=+1890.263667839" Oct 30 05:15:17.353565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:17.353507 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" podStartSLOduration=2.353489353 podStartE2EDuration="2.353489353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:17.348327522 +0000 UTC m=+1890.317822509" watchObservedRunningTime="2026-10-30 05:15:17.353489353 +0000 UTC m=+1890.322984336" Oct 30 05:15:19.277467 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:19.277423 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:15:19.664301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.664265 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:15:19.671514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.671482 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.688754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.688725 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:15:19.696071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.696041 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f8eb4\"/\"openshift-service-ca.crt\"" Oct 30 05:15:19.696071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.696047 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f8eb4\"/\"default-dockercfg-cqxv2\"" Oct 30 05:15:19.696071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.696058 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f8eb4\"/\"kube-root-ca.crt\"" Oct 30 05:15:19.703322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.703303 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f8eb4\"/\"test-model-1f8eb4-kserve-self-signed-certs\"" Oct 30 05:15:19.721973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.721935 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.722123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.721988 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.722123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.722068 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.722123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.722095 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.722349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.722180 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.722349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.722241 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhgb\" (UniqueName: \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823184 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823256 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhgb\" (UniqueName: \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823484 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823525 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823624 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823675 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823702 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.823827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.823788 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.824087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.824069 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.825913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.825885 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.826384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.826362 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:19.843529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:19.843507 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhgb\" (UniqueName: \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\") pod \"test-model-1f8eb4-kserve-664d44bcf9-jzjhs\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:20.007294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:20.004098 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:20.595860 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:20.595816 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70894cad_da3f_42f3_9935_5058bea692b4.slice/crio-e207ad1b4463a3517b4969ff86b26fcd98496cdc3b15d37febdf468bc3251174 WatchSource:0}: Error finding container e207ad1b4463a3517b4969ff86b26fcd98496cdc3b15d37febdf468bc3251174: Status 404 returned error can't find the container with id e207ad1b4463a3517b4969ff86b26fcd98496cdc3b15d37febdf468bc3251174 Oct 30 05:15:20.597084 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:20.597057 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:15:21.309138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:21.309053 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" event={"ID":"70894cad-da3f-42f3-9935-5058bea692b4","Type":"ContainerStarted","Data":"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c"} Oct 30 05:15:21.309138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:21.309101 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" event={"ID":"70894cad-da3f-42f3-9935-5058bea692b4","Type":"ContainerStarted","Data":"e207ad1b4463a3517b4969ff86b26fcd98496cdc3b15d37febdf468bc3251174"} Oct 30 05:15:22.005252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:22.005183 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:28.240017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:28.239986 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:15:28.240454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:28.240060 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:15:28.285039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:28.284982 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" podStartSLOduration=9.28496743 podStartE2EDuration="9.28496743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:21.364939572 +0000 UTC m=+1894.334434574" watchObservedRunningTime="2026-10-30 05:15:28.28496743 +0000 UTC m=+1901.254462445" Oct 30 05:15:30.004945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:30.004911 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:30.017008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:30.016980 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:30.424714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:30.424687 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:15:32.431619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.431584 5058 generic.go:356] "Generic (PLEG): container finished" podID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerID="c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b" exitCode=0 Oct 30 05:15:32.432087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.431660 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerDied","Data":"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b"} Oct 30 05:15:32.432087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.432046 5058 scope.go:122] "RemoveContainer" containerID="c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b" Oct 30 05:15:32.433666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.433636 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:32.433819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.433716 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:32.433819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.433765 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:32.867561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:32.867482 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:15:33.458920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:33.458885 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerStarted","Data":"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef"} Oct 30 05:15:34.350304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:34.350272 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:15:34.352601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:34.352549 5058 patch_prober.go:29] interesting pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:15:34.352756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:34.352644 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 05:15:35.350376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:35.350318 5058 patch_prober.go:29] interesting pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:15:35.350897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:35.350424 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 05:15:36.346240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:36.343329 5058 patch_prober.go:29] interesting pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:15:36.346240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:36.343403 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 05:15:37.338242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:37.329551 5058 patch_prober.go:29] interesting pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:15:37.338242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:37.329621 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 05:15:38.347261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:38.342609 5058 patch_prober.go:29] interesting pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:15:38.347261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:38.342671 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 05:15:39.331602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:39.331572 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:15:39.332072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:39.332050 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:15:39.332709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:39.332689 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:15:49.191526 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:15:49.191480 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:15:54.750061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.750016 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:15:54.756733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.756710 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:54.773278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.773251 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:15:54.782178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.782145 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-2f6fed-serving-cert\"" Oct 30 05:15:54.907403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.907374 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8n56\" (UniqueName: \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:54.907611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.907440 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:54.907735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.907712 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:54.907802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:54.907774 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.013921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.013831 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.014205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.014183 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.015234 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.015194 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8n56\" (UniqueName: \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.015739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.015621 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.018424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.018397 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.043002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.042929 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8n56\" (UniqueName: \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.071238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.061678 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.094300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.086913 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.374494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.374413 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:55.692713 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:15:55.692685 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda39a40c_a60f_4b18_9107_eef438c08482.slice/crio-cd874a52d20bc1606b7740d75d0893be3dc356f2163d1397832ee13b71df2a16 WatchSource:0}: Error finding container cd874a52d20bc1606b7740d75d0893be3dc356f2163d1397832ee13b71df2a16: Status 404 returned error can't find the container with id cd874a52d20bc1606b7740d75d0893be3dc356f2163d1397832ee13b71df2a16 Oct 30 05:15:55.694006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.693982 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:15:55.694331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.694314 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:15:55.815387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:55.815346 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" event={"ID":"da39a40c-a60f-4b18-9107-eef438c08482","Type":"ContainerStarted","Data":"cd874a52d20bc1606b7740d75d0893be3dc356f2163d1397832ee13b71df2a16"} Oct 30 05:15:56.824592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:56.824559 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" event={"ID":"da39a40c-a60f-4b18-9107-eef438c08482","Type":"ContainerStarted","Data":"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc"} Oct 30 05:15:56.825014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:56.824630 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:15:56.934267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:15:56.934056 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" podStartSLOduration=2.538880265 podStartE2EDuration="2.934034914s" totalImagesPullingTime="395.154649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:56.916582024 +0000 UTC m=+1929.886077011" watchObservedRunningTime="2026-10-30 05:15:56.934034914 +0000 UTC m=+1929.903529904" Oct 30 05:16:02.838724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:02.838692 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:16:19.191952 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:16:19.191913 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:16:21.475124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:21.475074 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:16:21.475662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:21.475316 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerName="maas-api" containerID="cri-o://e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f" gracePeriod=30 Oct 30 05:16:21.630557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:21.630513 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.155:8443/health\": dial tcp 10.132.0.155:8443: connect: connection refused" Oct 30 05:16:21.933327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:21.933112 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:16:22.125366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.123940 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\" (UniqueName: \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\") pod \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " Oct 30 05:16:22.125366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.124196 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\") pod \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " Oct 30 05:16:22.125366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.124375 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\") pod \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " Oct 30 05:16:22.125366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.124508 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\") pod \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\" (UID: \"2c88fae8-438f-4e38-8c3f-58766f21e5e0\") " Oct 30 05:16:22.126776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.126746 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca" pod "2c88fae8-438f-4e38-8c3f-58766f21e5e0" (UID: "2c88fae8-438f-4e38-8c3f-58766f21e5e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:22.139278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.138816 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892" pod "2c88fae8-438f-4e38-8c3f-58766f21e5e0" (UID: "2c88fae8-438f-4e38-8c3f-58766f21e5e0"). InnerVolumeSpecName "kube-api-access-c7892". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:22.139278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.138875 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs" pod "2c88fae8-438f-4e38-8c3f-58766f21e5e0" (UID: "2c88fae8-438f-4e38-8c3f-58766f21e5e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:22.150315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.149336 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls" pod "2c88fae8-438f-4e38-8c3f-58766f21e5e0" (UID: "2c88fae8-438f-4e38-8c3f-58766f21e5e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:22.200929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.200895 5058 generic.go:356] "Generic (PLEG): container finished" podID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerID="e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f" exitCode=0 Oct 30 05:16:22.201104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.200945 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" Oct 30 05:16:22.201104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.200979 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" event={"ID":"2c88fae8-438f-4e38-8c3f-58766f21e5e0","Type":"ContainerDied","Data":"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f"} Oct 30 05:16:22.201104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.201035 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz" event={"ID":"2c88fae8-438f-4e38-8c3f-58766f21e5e0","Type":"ContainerDied","Data":"94d1b7fe3bc78639be8ce497175fc62f772e332e817fc090668b9f7fe7c47fee"} Oct 30 05:16:22.201104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.201064 5058 scope.go:122] "RemoveContainer" containerID="e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f" Oct 30 05:16:22.213992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.213968 5058 scope.go:122] "RemoveContainer" containerID="e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f" Oct 30 05:16:22.214429 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:16:22.214396 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f\": container with ID starting with e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f not found: ID does not exist" containerID="e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f" Oct 30 05:16:22.214575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.214439 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f"} err="failed to get container status \"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f\": rpc error: code = NotFound desc = could not find container \"e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f\": container with ID starting with e346d3b3cabf0a77aaa18c1fc6998dd4fedb6d9a88b3600a0484e7c9fd13da5f not found: ID does not exist" Oct 30 05:16:22.227458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.227431 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88fae8-438f-4e38-8c3f-58766f21e5e0-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:16:22.227604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.227463 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:16:22.227604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.227477 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88fae8-438f-4e38-8c3f-58766f21e5e0-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:16:22.227604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.227491 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7892\" (UniqueName: \"kubernetes.io/projected/2c88fae8-438f-4e38-8c3f-58766f21e5e0-kube-api-access-c7892\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:16:22.257538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.257504 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:16:22.261317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:22.261286 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c3486b-755f9b865c-gxdpz"] Oct 30 05:16:23.163581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:23.163550 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" path="/var/lib/kubelet/pods/2c88fae8-438f-4e38-8c3f-58766f21e5e0/volumes" Oct 30 05:16:24.515833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.515797 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:16:24.516699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.516682 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerName="maas-api" Oct 30 05:16:24.516764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.516701 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerName="maas-api" Oct 30 05:16:24.516838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.516829 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c88fae8-438f-4e38-8c3f-58766f21e5e0" containerName="maas-api" Oct 30 05:16:24.522073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.522053 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.539185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.539147 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:16:24.566022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.565996 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2a8f18-serving-cert\"" Oct 30 05:16:24.671014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.670975 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.671195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.671165 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.671292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.671282 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwpw\" (UniqueName: \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.671393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.671369 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.784502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.776283 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwpw\" (UniqueName: \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.784888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.784852 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.785318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.785275 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.785748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.785724 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.810616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.809605 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.814773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.814739 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.833285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.828768 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwpw\" (UniqueName: \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.833486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.833289 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:24.844241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:24.838513 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:25.179532 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:16:25.179494 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54931758_280e_4ce8_80d0_ca8a689d0ea7.slice/crio-fb2573b41ed1a9d192fc51cf3f88a4fcd2689ca23d1770b527d9feab75c6577e WatchSource:0}: Error finding container fb2573b41ed1a9d192fc51cf3f88a4fcd2689ca23d1770b527d9feab75c6577e: Status 404 returned error can't find the container with id fb2573b41ed1a9d192fc51cf3f88a4fcd2689ca23d1770b527d9feab75c6577e Oct 30 05:16:25.181352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:25.181327 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:16:25.225957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:25.225918 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" event={"ID":"54931758-280e-4ce8-80d0-ca8a689d0ea7","Type":"ContainerStarted","Data":"fb2573b41ed1a9d192fc51cf3f88a4fcd2689ca23d1770b527d9feab75c6577e"} Oct 30 05:16:26.235035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:26.234994 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" event={"ID":"54931758-280e-4ce8-80d0-ca8a689d0ea7","Type":"ContainerStarted","Data":"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd"} Oct 30 05:16:26.235569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:26.235068 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:26.279307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:26.279252 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" podStartSLOduration=1.9616429530000001 podStartE2EDuration="2.279238621s" totalImagesPullingTime="317.595668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:26.277745229 +0000 UTC m=+1959.247240216" watchObservedRunningTime="2026-10-30 05:16:26.279238621 +0000 UTC m=+1959.248733608" Oct 30 05:16:31.251403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.251360 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:16:31.259326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.259296 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.278937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.278907 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:16:31.287446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287410 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287471 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287617 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287660 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287695 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287788 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287817 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287903 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqdx\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.287969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.287933 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.288903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.288876 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-802b8a-openshift-default-dockercfg-ttwjq\"" Oct 30 05:16:31.389160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389123 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqdx\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389163 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389291 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389317 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389379 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389398 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389420 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389515 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.389563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389541 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.390027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.389990 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.390150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.390064 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.390286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.390263 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.390470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.390453 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.390565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.390485 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.392053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.392029 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.392154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.392068 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.411080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.411055 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqdx\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.412530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.412506 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\") pod \"e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:31.585151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:31.583896 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:32.052917 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:16:32.052888 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b35bca4_8f53_433c_ab58_16a17c7a7831.slice/crio-eac2a8b3d066487da2dbc34c38731abb5475f234f9c5097c42c3fc2a734c9d50 WatchSource:0}: Error finding container eac2a8b3d066487da2dbc34c38731abb5475f234f9c5097c42c3fc2a734c9d50: Status 404 returned error can't find the container with id eac2a8b3d066487da2dbc34c38731abb5475f234f9c5097c42c3fc2a734c9d50 Oct 30 05:16:32.054033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.054010 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:16:32.055763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.055734 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:32.055840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.055800 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:32.055840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.055832 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:32.260253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.256773 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:16:32.403647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.403608 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerStarted","Data":"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e"} Oct 30 05:16:32.403813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.403682 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerStarted","Data":"eac2a8b3d066487da2dbc34c38731abb5475f234f9c5097c42c3fc2a734c9d50"} Oct 30 05:16:32.463019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.462966 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podStartSLOduration=1.462949542 podStartE2EDuration="1.462949542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:32.462584079 +0000 UTC m=+1965.432079063" watchObservedRunningTime="2026-10-30 05:16:32.462949542 +0000 UTC m=+1965.432444530" Oct 30 05:16:32.586298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:32.586258 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:33.593838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:33.593810 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:34.419046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:34.419008 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:34.419911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:34.419892 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:16:43.333020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.332984 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:16:43.338316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.338291 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.355330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.355282 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-453511-openshift-default-dockercfg-j7lkc\"" Oct 30 05:16:43.373682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.373649 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:16:43.452302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452264 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452413 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452482 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jsr8\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452524 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452546 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452625 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452670 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452702 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.452747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.452740 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554305 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554408 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jsr8\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554473 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554526 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554622 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554653 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554702 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554752 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.554939 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.557964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.555539 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.559828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.558927 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.559828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.559181 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.559828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.559689 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.560090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.560066 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.580391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.580347 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.581958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.581142 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.583468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.583405 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jsr8\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.584493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.584473 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.655458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.655421 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:43.985452 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:16:43.985425 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f7a4ec_12b8_473b_9cfd_2caed4a2e346.slice/crio-197e487ffb1f52afea39267a5847ee78e1dd1aa0befb898e8cca5c61ac18983e WatchSource:0}: Error finding container 197e487ffb1f52afea39267a5847ee78e1dd1aa0befb898e8cca5c61ac18983e: Status 404 returned error can't find the container with id 197e487ffb1f52afea39267a5847ee78e1dd1aa0befb898e8cca5c61ac18983e Oct 30 05:16:43.986191 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.986167 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:16:43.988852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.988651 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:43.988945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.988883 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:43.989012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:43.988986 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:44.547177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:44.547142 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" event={"ID":"17f7a4ec-12b8-473b-9cfd-2caed4a2e346","Type":"ContainerStarted","Data":"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d"} Oct 30 05:16:44.547628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:44.547197 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" event={"ID":"17f7a4ec-12b8-473b-9cfd-2caed4a2e346","Type":"ContainerStarted","Data":"197e487ffb1f52afea39267a5847ee78e1dd1aa0befb898e8cca5c61ac18983e"} Oct 30 05:16:44.595711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:44.595653 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podStartSLOduration=1.595639686 podStartE2EDuration="1.595639686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:44.59287518 +0000 UTC m=+1977.562370191" watchObservedRunningTime="2026-10-30 05:16:44.595639686 +0000 UTC m=+1977.565134674" Oct 30 05:16:44.655960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:44.655924 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:45.175376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.175340 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:16:45.180469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.180445 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.203193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.203167 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:16:45.205453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.205428 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e16297-serving-cert\"" Oct 30 05:16:45.281780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.281731 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.282210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.282184 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.282906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.282508 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.283151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.283133 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8264\" (UniqueName: \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.384853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.384815 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.385019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.384872 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.385019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.384909 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8264\" (UniqueName: \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.385116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.385069 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.385595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.385568 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.387375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.387354 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.387476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.387418 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.406792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.406770 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8264\" (UniqueName: \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\") pod \"maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.505561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.505025 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:45.659464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.659432 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:45.659730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.659493 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:45.892849 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:16:45.892809 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6842bad2_fc0e_49fb_9e5f_649ff4ef60a2.slice/crio-3b76b70f82f99e9342f74b1f10b77a78269a740fd41793c4b89270b5d83640d5 WatchSource:0}: Error finding container 3b76b70f82f99e9342f74b1f10b77a78269a740fd41793c4b89270b5d83640d5: Status 404 returned error can't find the container with id 3b76b70f82f99e9342f74b1f10b77a78269a740fd41793c4b89270b5d83640d5 Oct 30 05:16:45.894251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:45.894203 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:16:46.597241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:46.592309 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" event={"ID":"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2","Type":"ContainerStarted","Data":"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4"} Oct 30 05:16:46.597241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:46.592379 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" event={"ID":"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2","Type":"ContainerStarted","Data":"3b76b70f82f99e9342f74b1f10b77a78269a740fd41793c4b89270b5d83640d5"} Oct 30 05:16:46.597241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:46.592432 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:16:46.667241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:46.663531 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:46.667241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:46.663591 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:47.655956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:47.655927 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:47.656123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:47.655987 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:48.656543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:48.656506 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:48.656932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:48.656562 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:49.203891 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:16:49.203849 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:16:49.655967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:49.655934 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:49.656135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:49.655995 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:50.656802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:50.656769 5058 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:16:50.657357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:50.656822 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 05:16:51.672674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:51.670543 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:51.672674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:51.672359 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:51.676781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:51.676752 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:16:51.750337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:51.750277 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" podStartSLOduration=6.289443222 podStartE2EDuration="6.750261395s" totalImagesPullingTime="460.818173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:46.641473036 +0000 UTC m=+1979.610968026" watchObservedRunningTime="2026-10-30 05:16:51.750261395 +0000 UTC m=+1984.719756383" Oct 30 05:16:53.603348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:16:53.603318 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:17:15.621012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.620975 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:17:15.626922 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.626894 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.648192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.648161 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:17:15.654261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.654209 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ccafe0\"/\"default-dockercfg-mf4bp\"" Oct 30 05:17:15.654437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.654283 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ccafe0\"/\"test-model-ccafe0-kserve-self-signed-certs\"" Oct 30 05:17:15.688483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.688446 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ccafe0\"/\"kube-root-ca.crt\"" Oct 30 05:17:15.688674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.688487 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ccafe0\"/\"openshift-service-ca.crt\"" Oct 30 05:17:15.746681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.746642 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.746867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.746843 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.746952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.746938 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.746988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.746977 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgbx\" (UniqueName: \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.747074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.747059 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.747136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.747121 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.848800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.848764 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.848962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.848889 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.848962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.848938 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849068 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgbx\" (UniqueName: \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849143 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849166 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849207 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849276 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.849629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.849603 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.851184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.851156 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.851443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.851426 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.869143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.869117 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgbx\" (UniqueName: \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\") pod \"test-model-ccafe0-kserve-85d5b9ffc9-c6rql\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:15.951259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:15.948796 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:16.375468 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:17:16.375395 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7cd480_6be8_4564_9f8e_86bc3592133f.slice/crio-8e0e03076efa79f3eebd869887272ebb7095fdbe0ad798b58529a87893fb3612 WatchSource:0}: Error finding container 8e0e03076efa79f3eebd869887272ebb7095fdbe0ad798b58529a87893fb3612: Status 404 returned error can't find the container with id 8e0e03076efa79f3eebd869887272ebb7095fdbe0ad798b58529a87893fb3612 Oct 30 05:17:16.379597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:16.379567 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:17:16.987112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:16.987075 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" event={"ID":"ef7cd480-6be8-4564-9f8e-86bc3592133f","Type":"ContainerStarted","Data":"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d"} Oct 30 05:17:16.987540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:16.987120 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" event={"ID":"ef7cd480-6be8-4564-9f8e-86bc3592133f","Type":"ContainerStarted","Data":"8e0e03076efa79f3eebd869887272ebb7095fdbe0ad798b58529a87893fb3612"} Oct 30 05:17:17.032553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:17.032502 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" podStartSLOduration=2.032488656 podStartE2EDuration="2.032488656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:17.028539899 +0000 UTC m=+2009.998034885" watchObservedRunningTime="2026-10-30 05:17:17.032488656 +0000 UTC m=+2010.001983643" Oct 30 05:17:17.950785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:17.950754 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:19.204544 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:17:19.204507 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:17:25.951454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:25.951416 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:25.964508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:25.964477 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:26.163285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:26.163246 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:17:49.191073 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:17:49.191037 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:17:55.907917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:55.907883 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:17:55.908420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:17:55.908088 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" containerID="cri-o://bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45" gracePeriod=30 Oct 30 05:18:00.645318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:00.645282 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:18:00.645857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:00.645358 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:18:01.277173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.277147 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:18:01.444227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444181 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444263 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444330 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444359 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444386 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444623 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:01.444657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444605 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.444758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444739 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444850 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.444940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444857 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.444940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444909 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.445059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.444955 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\") pod \"a2a37256-910f-486f-855b-fbb2e850315d\" (UID: \"a2a37256-910f-486f-855b-fbb2e850315d\") " Oct 30 05:18:01.445115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445066 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.445382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445358 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.445842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445819 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a37256-910f-486f-855b-fbb2e850315d-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.445959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445848 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.445959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445861 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.445959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445872 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.445959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.445884 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.451371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.451325 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "kube-api-access-6qk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:01.451730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.451702 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:01.457914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.457882 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.458290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.458269 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token" pod "a2a37256-910f-486f-855b-fbb2e850315d" (UID: "a2a37256-910f-486f-855b-fbb2e850315d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:01.518666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.518589 5058 generic.go:356] "Generic (PLEG): container finished" podID="a2a37256-910f-486f-855b-fbb2e850315d" containerID="bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45" exitCode=0 Oct 30 05:18:01.518666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.518654 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" Oct 30 05:18:01.518881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.518670 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" event={"ID":"a2a37256-910f-486f-855b-fbb2e850315d","Type":"ContainerDied","Data":"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45"} Oct 30 05:18:01.518881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.518778 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q" event={"ID":"a2a37256-910f-486f-855b-fbb2e850315d","Type":"ContainerDied","Data":"2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb"} Oct 30 05:18:01.518881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.518813 5058 scope.go:122] "RemoveContainer" containerID="bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45" Oct 30 05:18:01.531130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.531107 5058 scope.go:122] "RemoveContainer" containerID="bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45" Oct 30 05:18:01.531559 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:01.531534 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45\": container with ID starting with bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45 not found: ID does not exist" containerID="bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45" Oct 30 05:18:01.531630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.531569 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45"} err="failed to get container status \"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45\": rpc error: code = NotFound desc = could not find container \"bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45\": container with ID starting with bfd56d64bfa5695a2ca41dfb0717d775eb148529976bca7f3dc735e0dffc3a45 not found: ID does not exist" Oct 30 05:18:01.546841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.546811 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qk5x\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-kube-api-access-6qk5x\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.546841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.546838 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a37256-910f-486f-855b-fbb2e850315d-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.546976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.546853 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a37256-910f-486f-855b-fbb2e850315d-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.546976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.546864 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a37256-910f-486f-855b-fbb2e850315d-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.577366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.577342 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:18:01.585736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:01.585706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q"] Oct 30 05:18:01.603369 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:01.603341 5058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a37256_910f_486f_855b_fbb2e850315d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a37256_910f_486f_855b_fbb2e850315d.slice/crio-2f419728c6d68f505617aa3d516cdb1907e48e4cf6f96b2e2552f37c113dcacb\": RecentStats: unable to find data in memory cache]" Oct 30 05:18:03.163109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:03.163072 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a37256-910f-486f-855b-fbb2e850315d" path="/var/lib/kubelet/pods/a2a37256-910f-486f-855b-fbb2e850315d/volumes" Oct 30 05:18:10.271684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:10.271654 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 05:18:10.272137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:10.271873 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" containerID="cri-o://7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1" gracePeriod=30 Oct 30 05:18:11.967011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:11.966979 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:18:15.559820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.559757 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 05:18:15.639477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639444 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.639669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639505 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.639669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639602 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.639806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639791 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.639862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639835 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:15.639964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.639946 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.640028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640014 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640131 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640141 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640194 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640300 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\") pod \"23f593cb-336b-40d8-a8cc-e4934a1109dd\" (UID: \"23f593cb-336b-40d8-a8cc-e4934a1109dd\") " Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640434 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640454 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.640496 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.641035 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.641056 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.641072 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.641084 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.641122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.641097 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f593cb-336b-40d8-a8cc-e4934a1109dd-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.642339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.642303 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:15.642885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.642863 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:15.643051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.643033 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:15.643972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.643944 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc" pod "23f593cb-336b-40d8-a8cc-e4934a1109dd" (UID: "23f593cb-336b-40d8-a8cc-e4934a1109dd"). InnerVolumeSpecName "kube-api-access-7s9vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:15.654194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.654168 5058 generic.go:356] "Generic (PLEG): container finished" podID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerID="7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1" exitCode=0 Oct 30 05:18:15.654335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.654229 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" Oct 30 05:18:15.654335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.654257 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerDied","Data":"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1"} Oct 30 05:18:15.654335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.654319 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx" event={"ID":"23f593cb-336b-40d8-a8cc-e4934a1109dd","Type":"ContainerDied","Data":"a72a870819bf8ae98d101c5609d85ecf054b51e58e6e694115d55780497311bd"} Oct 30 05:18:15.654467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.654340 5058 scope.go:122] "RemoveContainer" containerID="7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1" Oct 30 05:18:15.694088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.694063 5058 scope.go:122] "RemoveContainer" containerID="82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42" Oct 30 05:18:15.703995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.703974 5058 scope.go:122] "RemoveContainer" containerID="7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1" Oct 30 05:18:15.704292 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:15.704269 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1\": container with ID starting with 7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1 not found: ID does not exist" containerID="7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1" Oct 30 05:18:15.704384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.704298 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1"} err="failed to get container status \"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1\": rpc error: code = NotFound desc = could not find container \"7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1\": container with ID starting with 7a6e7e95447ed54a888f32d8360cec0f443e4682f536c480d96c12efff8c45c1 not found: ID does not exist" Oct 30 05:18:15.704384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.704316 5058 scope.go:122] "RemoveContainer" containerID="82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42" Oct 30 05:18:15.704553 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:15.704537 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42\": container with ID starting with 82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42 not found: ID does not exist" containerID="82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42" Oct 30 05:18:15.704602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.704556 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42"} err="failed to get container status \"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42\": rpc error: code = NotFound desc = could not find container \"82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42\": container with ID starting with 82e56a32f815f3d20c1a7514281145f7c660a41a543b0f7e548b707b4816ed42 not found: ID does not exist" Oct 30 05:18:15.736164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.736129 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 05:18:15.740191 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.740166 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx"] Oct 30 05:18:15.742836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.742813 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.742836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.742839 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.742966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.742848 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9vc\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-kube-api-access-7s9vc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:15.742966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:15.742857 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f593cb-336b-40d8-a8cc-e4934a1109dd-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:17.199082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:17.199052 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" path="/var/lib/kubelet/pods/23f593cb-336b-40d8-a8cc-e4934a1109dd/volumes" Oct 30 05:18:19.192078 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:19.192042 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:18:26.990801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:26.990741 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" podUID="70894cad-da3f-42f3-9935-5058bea692b4" containerName="main" containerID="cri-o://4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c" gracePeriod=45 Oct 30 05:18:27.308385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.308358 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:18:27.421380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421345 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421418 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421520 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421584 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421647 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\" (UniqueName: \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421692 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\") pod \"70894cad-da3f-42f3-9935-5058bea692b4\" (UID: \"70894cad-da3f-42f3-9935-5058bea692b4\") " Oct 30 05:18:27.421869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421731 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:27.421869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421764 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:27.421978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.421875 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:27.422341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.422317 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.422341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.422343 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.422527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.422354 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.423520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.423495 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:27.423685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.423664 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "kube-api-access-6bhgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:27.423764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.423744 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs" pod "70894cad-da3f-42f3-9935-5058bea692b4" (UID: "70894cad-da3f-42f3-9935-5058bea692b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:27.523942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.523864 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70894cad-da3f-42f3-9935-5058bea692b4-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.523942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.523895 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhgb\" (UniqueName: \"kubernetes.io/projected/70894cad-da3f-42f3-9935-5058bea692b4-kube-api-access-6bhgb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.523942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.523906 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70894cad-da3f-42f3-9935-5058bea692b4-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:27.814465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.814385 5058 generic.go:356] "Generic (PLEG): container finished" podID="70894cad-da3f-42f3-9935-5058bea692b4" containerID="4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c" exitCode=0 Oct 30 05:18:27.814465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.814433 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" Oct 30 05:18:27.814646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.814468 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" event={"ID":"70894cad-da3f-42f3-9935-5058bea692b4","Type":"ContainerDied","Data":"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c"} Oct 30 05:18:27.814646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.814515 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs" event={"ID":"70894cad-da3f-42f3-9935-5058bea692b4","Type":"ContainerDied","Data":"e207ad1b4463a3517b4969ff86b26fcd98496cdc3b15d37febdf468bc3251174"} Oct 30 05:18:27.814646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.814534 5058 scope.go:122] "RemoveContainer" containerID="4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c" Oct 30 05:18:27.825984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.825967 5058 scope.go:122] "RemoveContainer" containerID="4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c" Oct 30 05:18:27.826295 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:27.826272 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c\": container with ID starting with 4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c not found: ID does not exist" containerID="4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c" Oct 30 05:18:27.826359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.826304 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c"} err="failed to get container status \"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c\": rpc error: code = NotFound desc = could not find container \"4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c\": container with ID starting with 4f74e120a8cecf6f0e781045385069a4a8447e9aa72ed2a9138de7d05104964c not found: ID does not exist" Oct 30 05:18:27.871750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.871718 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:18:27.915120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:27.915085 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f8eb4/test-model-1f8eb4-kserve-664d44bcf9-jzjhs"] Oct 30 05:18:29.208539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:29.205909 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70894cad-da3f-42f3-9935-5058bea692b4" path="/var/lib/kubelet/pods/70894cad-da3f-42f3-9935-5058bea692b4/volumes" Oct 30 05:18:32.244936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.240936 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247710 5058 cpu_manager.go:414] "containerMap: removing container" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247756 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247818 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247826 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247849 5058 cpu_manager.go:414] "containerMap: removing container" podUID="70894cad-da3f-42f3-9935-5058bea692b4" containerName="main" Oct 30 05:18:32.247897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.247858 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70894cad-da3f-42f3-9935-5058bea692b4" containerName="main" Oct 30 05:18:32.248778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.248199 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:32.248778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.248242 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a37256-910f-486f-855b-fbb2e850315d" containerName="istio-proxy" Oct 30 05:18:32.248778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.248257 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="70894cad-da3f-42f3-9935-5058bea692b4" containerName="main" Oct 30 05:18:32.256341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.256315 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.269124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.269102 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:18:32.284765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.277794 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-a914e9-openshift-default-dockercfg-zl54b\"" Oct 30 05:18:32.403575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403589 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403696 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403760 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403818 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403930 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.403979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.403965 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrqr\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.404072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.404060 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.404117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.404087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505546 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505612 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505650 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505703 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505767 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.505975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505848 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.505980 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506032 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrqr\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506131 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506145 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506207 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506343 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506505 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.506787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.506771 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.508059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.508038 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.508313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.508290 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.526203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.526182 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.526337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.526291 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrqr\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.572967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.572935 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:32.930414 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:18:32.930375 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cec3e6f_9968_4ac4_94a2_0bc4b5b5d049.slice/crio-2d99daa67a35319382b8ac7a9c643373d854ecdcd4f6a720a4f3ac93c4b09782 WatchSource:0}: Error finding container 2d99daa67a35319382b8ac7a9c643373d854ecdcd4f6a720a4f3ac93c4b09782: Status 404 returned error can't find the container with id 2d99daa67a35319382b8ac7a9c643373d854ecdcd4f6a720a4f3ac93c4b09782 Oct 30 05:18:32.931805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.931777 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:18:32.933155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.933125 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:32.933260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.933185 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:32.933260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:32.933227 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:33.880166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:33.880122 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" event={"ID":"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049","Type":"ContainerStarted","Data":"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96"} Oct 30 05:18:33.880692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:33.880245 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" event={"ID":"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049","Type":"ContainerStarted","Data":"2d99daa67a35319382b8ac7a9c643373d854ecdcd4f6a720a4f3ac93c4b09782"} Oct 30 05:18:33.948660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:33.948537 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podStartSLOduration=1.948515912 podStartE2EDuration="1.948515912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:33.94256671 +0000 UTC m=+2086.912061700" watchObservedRunningTime="2026-10-30 05:18:33.948515912 +0000 UTC m=+2086.918010901" Oct 30 05:18:34.573381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:34.573345 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:34.575318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:34.575289 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:18:34.575449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:34.575360 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 05:18:35.573711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:35.573673 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:18:35.574093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:35.573723 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 05:18:36.058559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:36.058524 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:18:36.573493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:36.573461 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:18:36.573652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:36.573529 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 05:18:37.574301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:37.574265 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:18:37.574756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:37.574333 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 05:18:38.573706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:38.573671 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:18:38.573869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:38.573736 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 05:18:39.578429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:39.578402 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:39.578917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:39.578896 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:39.579559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:39.579539 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:18:41.059305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:41.059268 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" containerID="cri-o://412952d75c97fc2bcf82ecd332c23208c8651582806203904dc50c1cf5543e10" gracePeriod=25 Oct 30 05:18:41.379314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:41.379181 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:18:41.379480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:41.379433 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" podUID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" containerName="maas-api" containerID="cri-o://986175b0702d21b82ddef8ef76b3ff4ac42609cd3d62a4fc7d8ac16e05bc3ef9" gracePeriod=30 Oct 30 05:18:42.006521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.005432 5058 generic.go:356] "Generic (PLEG): container finished" podID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" containerID="986175b0702d21b82ddef8ef76b3ff4ac42609cd3d62a4fc7d8ac16e05bc3ef9" exitCode=0 Oct 30 05:18:42.006521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.005595 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" event={"ID":"a44169c0-3ba7-4d92-b0b0-ddeedace2887","Type":"ContainerDied","Data":"986175b0702d21b82ddef8ef76b3ff4ac42609cd3d62a4fc7d8ac16e05bc3ef9"} Oct 30 05:18:42.174928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.174898 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:18:42.225914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.225881 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\") pod \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " Oct 30 05:18:42.226068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.225977 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\") pod \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " Oct 30 05:18:42.226208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.226190 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\") pod \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " Oct 30 05:18:42.226328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.226275 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\" (UniqueName: \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\") pod \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\" (UID: \"a44169c0-3ba7-4d92-b0b0-ddeedace2887\") " Oct 30 05:18:42.226581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.226554 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca" pod "a44169c0-3ba7-4d92-b0b0-ddeedace2887" (UID: "a44169c0-3ba7-4d92-b0b0-ddeedace2887"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:42.227007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.226977 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44169c0-3ba7-4d92-b0b0-ddeedace2887-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:42.228409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.228368 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls" pod "a44169c0-3ba7-4d92-b0b0-ddeedace2887" (UID: "a44169c0-3ba7-4d92-b0b0-ddeedace2887"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:42.228587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.228569 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs" pod "a44169c0-3ba7-4d92-b0b0-ddeedace2887" (UID: "a44169c0-3ba7-4d92-b0b0-ddeedace2887"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:42.228853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.228821 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj" pod "a44169c0-3ba7-4d92-b0b0-ddeedace2887" (UID: "a44169c0-3ba7-4d92-b0b0-ddeedace2887"). InnerVolumeSpecName "kube-api-access-9bpmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:42.336077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.330351 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:42.336077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.330386 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpmj\" (UniqueName: \"kubernetes.io/projected/a44169c0-3ba7-4d92-b0b0-ddeedace2887-kube-api-access-9bpmj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:42.336077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:42.330399 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44169c0-3ba7-4d92-b0b0-ddeedace2887-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:43.014213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.014187 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" Oct 30 05:18:43.014400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.014232 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw" event={"ID":"a44169c0-3ba7-4d92-b0b0-ddeedace2887","Type":"ContainerDied","Data":"b974b98bce05b9f197c463e1a99776a8b95f0e3e437b3ccbd5be666bbde71921"} Oct 30 05:18:43.014400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.014294 5058 scope.go:122] "RemoveContainer" containerID="986175b0702d21b82ddef8ef76b3ff4ac42609cd3d62a4fc7d8ac16e05bc3ef9" Oct 30 05:18:43.070999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.070976 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:18:43.083868 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.083845 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f8eb4-ffcf585f5-lzsrw"] Oct 30 05:18:43.162945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:43.162916 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" path="/var/lib/kubelet/pods/a44169c0-3ba7-4d92-b0b0-ddeedace2887/volumes" Oct 30 05:18:47.605009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:47.604983 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:18:47.617809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:47.617789 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:18:49.047791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:49.047758 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:18:49.048285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:49.047985 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" containerID="cri-o://9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9" gracePeriod=30 Oct 30 05:18:49.373664 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:49.371452 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:18:49.886490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:49.886452 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:18:50.006265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.006209 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:18:50.007423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007405 5058 cpu_manager.go:414] "containerMap: removing container" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:50.007503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007426 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:50.007503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007448 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" containerName="maas-api" Oct 30 05:18:50.007503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007456 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" containerName="maas-api" Oct 30 05:18:50.007683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007672 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f593cb-336b-40d8-a8cc-e4934a1109dd" containerName="istio-proxy" Oct 30 05:18:50.007719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.007687 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44169c0-3ba7-4d92-b0b0-ddeedace2887" containerName="maas-api" Oct 30 05:18:50.014066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.014046 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.029369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.029346 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:18:50.039178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.039153 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8afb26-openshift-default-dockercfg-bnnnm\"" Oct 30 05:18:50.125597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125561 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.125952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125628 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgj6\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.125952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125783 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.125952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125839 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.126057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125946 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.126057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.125982 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.126127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.126068 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.126127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.126092 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.126212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.126137 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.227836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.227801 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.227989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.227861 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228190 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228283 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgj6\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228358 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228380 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228413 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228469 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228477 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228525 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228584 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228729 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.228907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.228879 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.229107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.229085 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.230687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.230667 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.231040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.231024 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.247999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.247964 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.248183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.248160 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgj6\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.329527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.329489 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:50.647516 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:18:50.647485 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b0bc50_eba6_468e_a81e_1deb86c601ab.slice/crio-2abbd1faac12edca833022c02551bfa7b8964ca331e488209b946eae92622f8f WatchSource:0}: Error finding container 2abbd1faac12edca833022c02551bfa7b8964ca331e488209b946eae92622f8f: Status 404 returned error can't find the container with id 2abbd1faac12edca833022c02551bfa7b8964ca331e488209b946eae92622f8f Oct 30 05:18:50.650091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.650067 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:18:50.650332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.650277 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:50.650388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.650376 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:50.650423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:50.650405 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:51.101120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.101036 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" event={"ID":"13b0bc50-eba6-468e-a81e-1deb86c601ab","Type":"ContainerStarted","Data":"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16"} Oct 30 05:18:51.101120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.101095 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" event={"ID":"13b0bc50-eba6-468e-a81e-1deb86c601ab","Type":"ContainerStarted","Data":"2abbd1faac12edca833022c02551bfa7b8964ca331e488209b946eae92622f8f"} Oct 30 05:18:51.150758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.150702 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podStartSLOduration=2.150686624 podStartE2EDuration="2.150686624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:51.148485752 +0000 UTC m=+2104.117980753" watchObservedRunningTime="2026-10-30 05:18:51.150686624 +0000 UTC m=+2104.120181612" Oct 30 05:18:51.338979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.338945 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:51.344535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.344498 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:51.346798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:51.346765 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:52.349411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:52.349377 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:52.350043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:52.350007 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:53.346268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:53.346234 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:53.346539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:53.346510 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:54.354907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.354820 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:54.356665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.356617 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:54.396388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.396358 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:18:54.553882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.553843 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.554475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.554447 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.554975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.554866 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.555431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555343 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.555431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555408 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.555578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555501 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.555578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555558 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.555681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555617 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.555729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555687 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.555782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555736 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.556548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.555918 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\") pod \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\" (UID: \"2b69f1ca-de52-471b-a19d-8be6f8914d6a\") " Oct 30 05:18:54.556548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.556068 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.556548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.556246 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.557175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557140 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:54.557377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557358 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.557457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557404 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.557457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557420 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.557457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557433 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.557457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.557446 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.559074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.559042 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.561167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.561143 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:54.561268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.561241 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.561593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.561568 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl" pod "2b69f1ca-de52-471b-a19d-8be6f8914d6a" (UID: "2b69f1ca-de52-471b-a19d-8be6f8914d6a"). InnerVolumeSpecName "kube-api-access-nlqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.658844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.658799 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.658844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.658838 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.658844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.658853 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.659168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:54.658867 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqxl\" (UniqueName: \"kubernetes.io/projected/2b69f1ca-de52-471b-a19d-8be6f8914d6a-kube-api-access-nlqxl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:18:55.173978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.173935 5058 generic.go:356] "Generic (PLEG): container finished" podID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerID="9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9" exitCode=0 Oct 30 05:18:55.174182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.174085 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:18:55.177608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.177574 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" event={"ID":"2b69f1ca-de52-471b-a19d-8be6f8914d6a","Type":"ContainerDied","Data":"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9"} Oct 30 05:18:55.177837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.177814 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" event={"ID":"2b69f1ca-de52-471b-a19d-8be6f8914d6a","Type":"ContainerDied","Data":"72bfde4dc9b9d95ef63b49105f21de7987a2f108552d454c78c87c8178cf0271"} Oct 30 05:18:55.177975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.177961 5058 scope.go:122] "RemoveContainer" containerID="9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9" Oct 30 05:18:55.197670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.197652 5058 scope.go:122] "RemoveContainer" containerID="9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9" Oct 30 05:18:55.198075 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:18:55.198036 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9\": container with ID starting with 9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9 not found: ID does not exist" containerID="9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9" Oct 30 05:18:55.198173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.198070 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9"} err="failed to get container status \"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9\": rpc error: code = NotFound desc = could not find container \"9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9\": container with ID starting with 9e9ac1cf04271de958fc9408a5ec024b52456471493481acc03d08cb8f2dd5e9 not found: ID does not exist" Oct 30 05:18:55.336306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.335579 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:55.336306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:55.335652 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:56.336361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:56.336315 5058 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:18:56.338416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:56.338386 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 05:18:57.335607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:57.335576 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:57.336051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:57.336034 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:57.336829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:57.336797 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:18:58.323965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.323889 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:18:58.325007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.324984 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" Oct 30 05:18:58.325007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.325006 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" Oct 30 05:18:58.325306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.325290 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" containerName="istio-proxy" Oct 30 05:18:58.330678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.330654 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.351161 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.351137 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:18:58.378609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.365602 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e16297-openshift-default-dockercfg-gzhs5\"" Oct 30 05:18:58.413093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.413062 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:18:58.417940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.417918 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.436847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.436819 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2f6fed\"/\"openshift-service-ca.crt\"" Oct 30 05:18:58.441547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.437121 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2f6fed\"/\"kube-root-ca.crt\"" Oct 30 05:18:58.441547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.439894 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441541 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441582 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441669 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441766 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l459r\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441800 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.441944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441907 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.442035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441966 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.442035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.441996 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.446269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.446239 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:18:58.447133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.447117 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2f6fed\"/\"auth-test-model-2f6fed-kserve-self-signed-certs\"" Oct 30 05:18:58.447197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.447166 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2f6fed\"/\"default-dockercfg-nk7bg\"" Oct 30 05:18:58.543348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543317 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543350 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.543542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543385 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j2t\" (UniqueName: \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.543542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543456 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543480 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543618 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.543745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543714 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543721 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543787 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.543896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543825 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543862 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543900 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.543997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543909 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543935 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.543997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.543949 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.544231 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.544200 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.544304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.544269 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.544374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.544350 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l459r\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.544479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.544445 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.544652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.544608 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.546278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.546248 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.546368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.546356 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.565170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.565143 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.565784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.565763 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l459r\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.679360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.671783 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.691492 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.691910 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.692197 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.692289 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.692353 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.692523 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.692564 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j2t\" (UniqueName: \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.701759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.701610 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.755245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.751243 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.796372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.796342 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.796668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.796641 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:58.803146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:58.803120 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j2t\" (UniqueName: \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\") pod \"auth-test-model-2f6fed-kserve-7ddf946457-8t56p\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:59.048986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.048911 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:18:59.107948 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:18:59.107913 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97d1c81_21d0_45e5_b4ab_054306d87115.slice/crio-cb919ea3d06b2ddc2df6ca5511377a40995e1c2096c8854e778b5f4dd8e8d0fd WatchSource:0}: Error finding container cb919ea3d06b2ddc2df6ca5511377a40995e1c2096c8854e778b5f4dd8e8d0fd: Status 404 returned error can't find the container with id cb919ea3d06b2ddc2df6ca5511377a40995e1c2096c8854e778b5f4dd8e8d0fd Oct 30 05:18:59.111970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.111130 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:18:59.111970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.111428 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:59.111970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.111518 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:59.111970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.111561 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:59.210049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.210010 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" event={"ID":"b97d1c81-21d0-45e5-b4ab-054306d87115","Type":"ContainerStarted","Data":"cb919ea3d06b2ddc2df6ca5511377a40995e1c2096c8854e778b5f4dd8e8d0fd"} Oct 30 05:18:59.369769 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:18:59.369743 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ea2a04_6082_4c07_99e5_8b0b70e24207.slice/crio-f56450b6d1631b927ee494cff904246a3f52f258825d00ae21d80f5e13dd6c8d WatchSource:0}: Error finding container f56450b6d1631b927ee494cff904246a3f52f258825d00ae21d80f5e13dd6c8d: Status 404 returned error can't find the container with id f56450b6d1631b927ee494cff904246a3f52f258825d00ae21d80f5e13dd6c8d Oct 30 05:18:59.371321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.371296 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:18:59.886153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:18:59.886113 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:00.147969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.147933 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:19:00.253241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.223585 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" event={"ID":"b97d1c81-21d0-45e5-b4ab-054306d87115","Type":"ContainerStarted","Data":"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e"} Oct 30 05:19:00.263798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.263631 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" event={"ID":"d1ea2a04-6082-4c07-99e5-8b0b70e24207","Type":"ContainerStarted","Data":"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9"} Oct 30 05:19:00.263798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.263684 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" event={"ID":"d1ea2a04-6082-4c07-99e5-8b0b70e24207","Type":"ContainerStarted","Data":"f56450b6d1631b927ee494cff904246a3f52f258825d00ae21d80f5e13dd6c8d"} Oct 30 05:19:00.281255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.280359 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podStartSLOduration=2.280345174 podStartE2EDuration="2.280345174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:00.276796613 +0000 UTC m=+2113.246291602" watchObservedRunningTime="2026-10-30 05:19:00.280345174 +0000 UTC m=+2113.249840161" Oct 30 05:19:00.322340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.322290 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" podStartSLOduration=2.322274252 podStartE2EDuration="2.322274252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:00.317686126 +0000 UTC m=+2113.287181114" watchObservedRunningTime="2026-10-30 05:19:00.322274252 +0000 UTC m=+2113.291769233" Oct 30 05:19:00.692259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.692190 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:19:00.694569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.694543 5058 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 05:19:00.694691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:00.694590 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 05:19:01.068316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.062612 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:19:01.286757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.285335 5058 generic.go:356] "Generic (PLEG): container finished" podID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerID="412952d75c97fc2bcf82ecd332c23208c8651582806203904dc50c1cf5543e10" exitCode=0 Oct 30 05:19:01.286757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.286381 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" event={"ID":"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f","Type":"ContainerDied","Data":"412952d75c97fc2bcf82ecd332c23208c8651582806203904dc50c1cf5543e10"} Oct 30 05:19:01.407013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.405585 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:19:01.555455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.555418 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\" (UniqueName: \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\") pod \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " Oct 30 05:19:01.555628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.555477 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\") pod \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\" (UID: \"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f\") " Oct 30 05:19:01.555860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.555836 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume" pod "c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" (UID: "c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:01.556053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.556029 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:01.557724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.557691 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk" pod "c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" (UID: "c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f"). InnerVolumeSpecName "kube-api-access-lc9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:01.657338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.657310 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9tk\" (UniqueName: \"kubernetes.io/projected/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f-kube-api-access-lc9tk\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:01.692131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.692101 5058 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 05:19:01.692301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:01.692149 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 05:19:02.326298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.323912 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" Oct 30 05:19:02.338243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.337734 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch" event={"ID":"c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f","Type":"ContainerDied","Data":"4a36edb52c8543d657de8f75af21c705b112459dfedeb367d36227f06fb97970"} Oct 30 05:19:02.338243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.337814 5058 scope.go:122] "RemoveContainer" containerID="412952d75c97fc2bcf82ecd332c23208c8651582806203904dc50c1cf5543e10" Oct 30 05:19:02.422214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.422177 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:19:02.443241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.438655 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch"] Oct 30 05:19:02.692991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.692955 5058 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 05:19:02.693415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:02.693009 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 05:19:03.163199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:03.163165 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" path="/var/lib/kubelet/pods/c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f/volumes" Oct 30 05:19:03.693062 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:03.693027 5058 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 05:19:03.693598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:03.693087 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 05:19:04.692695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:04.692654 5058 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 05:19:04.692882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:04.692727 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 05:19:05.148814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.148760 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" containerID="cri-o://988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716" gracePeriod=25 Oct 30 05:19:05.697197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.697168 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:19:05.697688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.697674 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:19:05.698409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.698386 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:19:05.925081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.925049 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:19:05.925995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.925972 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" Oct 30 05:19:05.925995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.925995 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" Oct 30 05:19:05.926212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.926197 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f9a72e-f69c-4eaf-9a58-906d2aae2e3f" containerName="payload-pre-processing" Oct 30 05:19:05.931759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.931739 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:05.948791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.948739 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:19:05.949144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:05.949072 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-68b893-serving-cert\"" Oct 30 05:19:06.084322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.084284 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lmm\" (UniqueName: \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.084663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.084640 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.084754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.084682 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.084822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.084807 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.187619 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.187677 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.187743 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.187790 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lmm\" (UniqueName: \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.189699 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:06.190305 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-serving-cert: secret "maas-api-e2e-inf-b-68b893-serving-cert" not found Oct 30 05:19:06.190682 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:06.190378 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls podName:789cc872-1991-457e-abf0-12e3eaea25c3 nodeName:}" failed. No retries permitted until 2026-10-30 05:19:06.690359991 +0000 UTC m=+2119.659854960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls") pod "maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" (UID: "789cc872-1991-457e-abf0-12e3eaea25c3") : secret "maas-api-e2e-inf-b-68b893-serving-cert" not found Oct 30 05:19:06.194343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.194319 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.213907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.213853 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lmm\" (UniqueName: \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.695046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.695014 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.697420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.697391 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:06.847942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:06.847907 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:07.292642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:07.292613 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:19:07.391274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:07.391209 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" event={"ID":"789cc872-1991-457e-abf0-12e3eaea25c3","Type":"ContainerStarted","Data":"64d75121c5c93fed51ac807a42789c2111b2a7e61422fa6c2d5d7f1d151ab565"} Oct 30 05:19:07.453820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:07.453782 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:08.401747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:08.401702 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" event={"ID":"789cc872-1991-457e-abf0-12e3eaea25c3","Type":"ContainerStarted","Data":"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9"} Oct 30 05:19:08.402187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:08.401778 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:08.443467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:08.443415 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" podStartSLOduration=2.898240287 podStartE2EDuration="3.443399433s" totalImagesPullingTime="545.159146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:08.442636365 +0000 UTC m=+2121.412131356" watchObservedRunningTime="2026-10-30 05:19:08.443399433 +0000 UTC m=+2121.412894422" Oct 30 05:19:09.049681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:09.049640 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:19:09.062411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:09.062389 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:19:09.423926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:09.423896 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:19:14.417379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:14.417349 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:19:17.272826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.272791 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:19:17.278553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.278527 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.308751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308724 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.308891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308761 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.308933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308893 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.308968 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308947 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.309002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kkd5\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.309002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.308988 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.309068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.309024 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.309068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.309062 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.309137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.309098 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.310816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.310785 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:19:17.410268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410203 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410371 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410393 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410464 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410506 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410526 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kkd5\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410545 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410635 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410832 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.410961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.410940 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.411095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.411074 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.411170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.411152 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.411253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.411185 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.412900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.412872 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.413134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.413117 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.430554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.430529 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.430822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.430803 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kkd5\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.451024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.451000 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:17.612137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.612062 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:17.935189 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:19:17.935163 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de475be_4827_4da5_8563_16ca8beb390b.slice/crio-50a2842108bf9c8be312499fb84ccbe033a4eb20754ee68b056242f0a86ff42d WatchSource:0}: Error finding container 50a2842108bf9c8be312499fb84ccbe033a4eb20754ee68b056242f0a86ff42d: Status 404 returned error can't find the container with id 50a2842108bf9c8be312499fb84ccbe033a4eb20754ee68b056242f0a86ff42d Oct 30 05:19:17.936302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.936275 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:19:17.938047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.938017 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:17.938125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.938080 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:17.938125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:17.938114 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:18.503530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:18.503472 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" event={"ID":"8de475be-4827-4da5-8563-16ca8beb390b","Type":"ContainerStarted","Data":"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759"} Oct 30 05:19:18.503991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:18.503569 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" event={"ID":"8de475be-4827-4da5-8563-16ca8beb390b","Type":"ContainerStarted","Data":"50a2842108bf9c8be312499fb84ccbe033a4eb20754ee68b056242f0a86ff42d"} Oct 30 05:19:18.576612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:18.576564 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podStartSLOduration=1.576551247 podStartE2EDuration="1.576551247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:18.559437792 +0000 UTC m=+2131.528932780" watchObservedRunningTime="2026-10-30 05:19:18.576551247 +0000 UTC m=+2131.546046233" Oct 30 05:19:18.613063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:18.613035 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:19.193871 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:19.193832 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:19:19.613525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:19.613442 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 05:19:19.613525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:19.613492 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 05:19:20.612750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:20.612714 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 05:19:20.612927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:20.612764 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 05:19:21.612841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:21.612808 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 05:19:21.613250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:21.612867 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 05:19:22.612839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:22.612803 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 05:19:22.613011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:22.612872 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 05:19:23.623331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:23.615880 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 05:19:23.623331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:23.615949 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 05:19:24.618206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:24.618176 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:24.618662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:24.618637 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:24.619175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:24.619153 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:19:25.187347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.187315 5058 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod2b69f1ca-de52-471b-a19d-8be6f8914d6a"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod2b69f1ca-de52-471b-a19d-8be6f8914d6a] : Timed out while waiting for systemd to remove kubepods-burstable-pod2b69f1ca_de52_471b_a19d_8be6f8914d6a.slice" Oct 30 05:19:25.187695 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:25.187354 5058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod2b69f1ca-de52-471b-a19d-8be6f8914d6a] : unable to destroy cgroup paths for cgroup [kubepods burstable pod2b69f1ca-de52-471b-a19d-8be6f8914d6a] : Timed out while waiting for systemd to remove kubepods-burstable-pod2b69f1ca_de52_471b_a19d_8be6f8914d6a.slice" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" Oct 30 05:19:25.524950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.524921 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:19:25.590764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.590730 5058 generic.go:356] "Generic (PLEG): container finished" podID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerID="988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716" exitCode=0 Oct 30 05:19:25.590945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.590787 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" Oct 30 05:19:25.590945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.590810 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" event={"ID":"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3","Type":"ContainerDied","Data":"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716"} Oct 30 05:19:25.590945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.590871 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc" event={"ID":"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3","Type":"ContainerDied","Data":"ca94b6834ed95b68cda6717974a5160bc3922d3334beb8cc178046643502669b"} Oct 30 05:19:25.590945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.590897 5058 scope.go:122] "RemoveContainer" containerID="988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716" Oct 30 05:19:25.591601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.591576 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7" Oct 30 05:19:25.603805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.603781 5058 scope.go:122] "RemoveContainer" containerID="988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716" Oct 30 05:19:25.604111 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:25.604088 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716\": container with ID starting with 988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716 not found: ID does not exist" containerID="988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716" Oct 30 05:19:25.604212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.604120 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716"} err="failed to get container status \"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716\": rpc error: code = NotFound desc = could not find container \"988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716\": container with ID starting with 988e2c8a3244fba2038b68bed39891aa4cd65e049413caac42187eb078b12716 not found: ID does not exist" Oct 30 05:19:25.648290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.648258 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:19:25.661156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.661133 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7"] Oct 30 05:19:25.697613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.697582 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\" (UniqueName: \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\") pod \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " Oct 30 05:19:25.697748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.697631 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\") pod \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\" (UID: \"7b2fb18e-b987-4fe5-8e93-845d5d0a50c3\") " Oct 30 05:19:25.698028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.698007 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume" pod "7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" (UID: "7b2fb18e-b987-4fe5-8e93-845d5d0a50c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:25.698349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.698330 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:25.699820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.699800 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw" pod "7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" (UID: "7b2fb18e-b987-4fe5-8e93-845d5d0a50c3"). InnerVolumeSpecName "kube-api-access-jpzkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:25.799523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.799458 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpzkw\" (UniqueName: \"kubernetes.io/projected/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3-kube-api-access-jpzkw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:25.941491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.941460 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:19:25.947167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:25.947142 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc"] Oct 30 05:19:27.220237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:27.216800 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b69f1ca-de52-471b-a19d-8be6f8914d6a" path="/var/lib/kubelet/pods/2b69f1ca-de52-471b-a19d-8be6f8914d6a/volumes" Oct 30 05:19:27.220237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:27.217551 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" path="/var/lib/kubelet/pods/7b2fb18e-b987-4fe5-8e93-845d5d0a50c3/volumes" Oct 30 05:19:37.743001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.742920 5058 generic.go:356] "Generic (PLEG): container finished" podID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerID="987fd5b39a006fc3bec357e67441f619d2b512064584ddf81dce0d939bf9e39e" exitCode=0 Oct 30 05:19:37.743472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.742994 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerDied","Data":"987fd5b39a006fc3bec357e67441f619d2b512064584ddf81dce0d939bf9e39e"} Oct 30 05:19:37.743545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.743477 5058 scope.go:122] "RemoveContainer" containerID="987fd5b39a006fc3bec357e67441f619d2b512064584ddf81dce0d939bf9e39e" Oct 30 05:19:37.745521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.745478 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:37.745618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.745580 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:37.745698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:37.745679 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:38.084645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.084548 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:19:38.084867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.084840 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerName="maas-api" containerID="cri-o://b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1" gracePeriod=30 Oct 30 05:19:38.751296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.731143 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:19:38.811253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.809113 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\" (UniqueName: \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\") pod \"05dbdf1d-648c-4b6e-a896-acddc715558c\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " Oct 30 05:19:38.811253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.809178 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") pod \"05dbdf1d-648c-4b6e-a896-acddc715558c\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " Oct 30 05:19:38.821382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.821342 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj" pod "05dbdf1d-648c-4b6e-a896-acddc715558c" (UID: "05dbdf1d-648c-4b6e-a896-acddc715558c"). InnerVolumeSpecName "kube-api-access-6r5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.826319 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\") pod \"05dbdf1d-648c-4b6e-a896-acddc715558c\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.826506 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\") pod \"05dbdf1d-648c-4b6e-a896-acddc715558c\" (UID: \"05dbdf1d-648c-4b6e-a896-acddc715558c\") " Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.827232 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5xj\" (UniqueName: \"kubernetes.io/projected/05dbdf1d-648c-4b6e-a896-acddc715558c-kube-api-access-6r5xj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.827706 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca" pod "05dbdf1d-648c-4b6e-a896-acddc715558c" (UID: "05dbdf1d-648c-4b6e-a896-acddc715558c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.830748 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs" pod "05dbdf1d-648c-4b6e-a896-acddc715558c" (UID: "05dbdf1d-648c-4b6e-a896-acddc715558c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:38.837670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.836868 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerStarted","Data":"3ec2f29dbbdc0fca0f0976c0fd4699c68ff35b92a5c8a35e9e9c0178b4aefe0e"} Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.839654 5058 generic.go:356] "Generic (PLEG): container finished" podID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerID="b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1" exitCode=0 Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.839697 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.839740 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" event={"ID":"05dbdf1d-648c-4b6e-a896-acddc715558c","Type":"ContainerDied","Data":"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1"} Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.839801 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" event={"ID":"05dbdf1d-648c-4b6e-a896-acddc715558c","Type":"ContainerDied","Data":"c37f03d14db381b84b03faf3323b87ab5a8d28686f273893027e35576f362219"} Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.839830 5058 scope.go:122] "RemoveContainer" containerID="b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1" Oct 30 05:19:38.844200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.841822 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls" pod "05dbdf1d-648c-4b6e-a896-acddc715558c" (UID: "05dbdf1d-648c-4b6e-a896-acddc715558c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:38.884572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.884538 5058 scope.go:122] "RemoveContainer" containerID="b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1" Oct 30 05:19:38.884901 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:38.884873 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1\": container with ID starting with b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1 not found: ID does not exist" containerID="b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1" Oct 30 05:19:38.884990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.884913 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1"} err="failed to get container status \"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1\": rpc error: code = NotFound desc = could not find container \"b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1\": container with ID starting with b7dad1f9c7f854afe8b87fcb38d5a15abae6f07ec2f18174ff54f9b498a952e1 not found: ID does not exist" Oct 30 05:19:38.932262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.931766 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:38.932262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.931803 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dbdf1d-648c-4b6e-a896-acddc715558c-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:38.932262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:38.931819 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dbdf1d-648c-4b6e-a896-acddc715558c-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:19:39.264271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:39.263471 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:19:39.269059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:39.269013 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr"] Oct 30 05:19:39.611263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:39.611156 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:19:39.611972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:39.611948 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 05:19:39.612067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:39.611996 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 05:19:40.611594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:40.611554 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 05:19:40.612002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:40.611622 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 05:19:41.168142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:41.168098 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" path="/var/lib/kubelet/pods/05dbdf1d-648c-4b6e-a896-acddc715558c/volumes" Oct 30 05:19:41.368279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:41.357411 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-36fd3c-68fd89ff87-ks4tr" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.163:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:19:41.610972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:41.610891 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 05:19:41.610972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:41.610959 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 05:19:42.049392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:42.049348 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:19:42.610690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:42.610656 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 05:19:42.610885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:42.610722 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 05:19:43.611160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:43.611129 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 05:19:43.611555 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:43.611208 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 05:19:44.367580 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:44.367546 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:19:44.403975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:44.403939 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:19:44.863640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:44.863585 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:19:44.866174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:44.866142 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:19:49.192020 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:19:49.191984 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:19:49.368656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:49.368620 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" containerID="cri-o://08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60" gracePeriod=25 Oct 30 05:19:49.404277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:49.404235 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" containerID="cri-o://562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b" gracePeriod=25 Oct 30 05:19:50.579575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:50.579536 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:50.580014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:19:50.579532 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:00.580472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:00.580426 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:00.580955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:00.580523 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:09.728972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.728946 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:20:09.732792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.732770 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.824838 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\" (UniqueName: \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\") pod \"5d0a6347-86f9-4734-adb7-71037d2b38cb\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.824929 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\" (UniqueName: \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\") pod \"88a85529-1cd0-42f4-be40-edefe3420042\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.825303 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\") pod \"88a85529-1cd0-42f4-be40-edefe3420042\" (UID: \"88a85529-1cd0-42f4-be40-edefe3420042\") " Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.825721 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume" pod "88a85529-1cd0-42f4-be40-edefe3420042" (UID: "88a85529-1cd0-42f4-be40-edefe3420042"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.825832 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\") pod \"5d0a6347-86f9-4734-adb7-71037d2b38cb\" (UID: \"5d0a6347-86f9-4734-adb7-71037d2b38cb\") " Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.826182 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume" pod "5d0a6347-86f9-4734-adb7-71037d2b38cb" (UID: "5d0a6347-86f9-4734-adb7-71037d2b38cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.826840 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a85529-1cd0-42f4-be40-edefe3420042-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.826859 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0a6347-86f9-4734-adb7-71037d2b38cb-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:09.824877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.828135 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl" pod "88a85529-1cd0-42f4-be40-edefe3420042" (UID: "88a85529-1cd0-42f4-be40-edefe3420042"). InnerVolumeSpecName "kube-api-access-kxdsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:09.830079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.830053 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw" pod "5d0a6347-86f9-4734-adb7-71037d2b38cb" (UID: "5d0a6347-86f9-4734-adb7-71037d2b38cb"). InnerVolumeSpecName "kube-api-access-74lqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:09.929111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.929024 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74lqw\" (UniqueName: \"kubernetes.io/projected/5d0a6347-86f9-4734-adb7-71037d2b38cb-kube-api-access-74lqw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:09.929111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:09.929052 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdsl\" (UniqueName: \"kubernetes.io/projected/88a85529-1cd0-42f4-be40-edefe3420042-kube-api-access-kxdsl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:10.150039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.150006 5058 generic.go:356] "Generic (PLEG): container finished" podID="88a85529-1cd0-42f4-be40-edefe3420042" containerID="562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b" exitCode=0 Oct 30 05:20:10.150290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.150071 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" Oct 30 05:20:10.150290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.150149 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" event={"ID":"88a85529-1cd0-42f4-be40-edefe3420042","Type":"ContainerDied","Data":"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b"} Oct 30 05:20:10.150290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.150197 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t" event={"ID":"88a85529-1cd0-42f4-be40-edefe3420042","Type":"ContainerDied","Data":"f4809bc0c6c1ad1c86c697ae5cd03b6a20a202e40fc033542d657db8db0986e9"} Oct 30 05:20:10.150290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.150269 5058 scope.go:122] "RemoveContainer" containerID="562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b" Oct 30 05:20:10.151635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.151609 5058 generic.go:356] "Generic (PLEG): container finished" podID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerID="08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60" exitCode=0 Oct 30 05:20:10.151748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.151664 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" Oct 30 05:20:10.151748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.151661 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" event={"ID":"5d0a6347-86f9-4734-adb7-71037d2b38cb","Type":"ContainerDied","Data":"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60"} Oct 30 05:20:10.151748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.151716 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn" event={"ID":"5d0a6347-86f9-4734-adb7-71037d2b38cb","Type":"ContainerDied","Data":"070d4a36ffe0259a0ca262cde960355f7f6e875d3a6750cb5253beb53ac533b9"} Oct 30 05:20:10.162539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.162523 5058 scope.go:122] "RemoveContainer" containerID="562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b" Oct 30 05:20:10.162790 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:10.162768 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b\": container with ID starting with 562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b not found: ID does not exist" containerID="562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b" Oct 30 05:20:10.162851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.162803 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b"} err="failed to get container status \"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b\": rpc error: code = NotFound desc = could not find container \"562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b\": container with ID starting with 562a591a9ba39d3b22c0ccd724055277e93e03bbc39d0020159ad6fe0ae4170b not found: ID does not exist" Oct 30 05:20:10.162851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.162827 5058 scope.go:122] "RemoveContainer" containerID="08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60" Oct 30 05:20:10.172411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.172385 5058 scope.go:122] "RemoveContainer" containerID="08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60" Oct 30 05:20:10.172666 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:10.172643 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60\": container with ID starting with 08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60 not found: ID does not exist" containerID="08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60" Oct 30 05:20:10.172774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.172673 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60"} err="failed to get container status \"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60\": rpc error: code = NotFound desc = could not find container \"08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60\": container with ID starting with 08b06dd5236c6e0b9e82d4318f4ed50c3f104e2bef411d0589800a94c2423c60 not found: ID does not exist" Oct 30 05:20:10.216776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.216703 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:20:10.220591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.220557 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn"] Oct 30 05:20:10.262306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.262272 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:20:10.269732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:10.269706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t"] Oct 30 05:20:11.212881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:11.212841 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" path="/var/lib/kubelet/pods/5d0a6347-86f9-4734-adb7-71037d2b38cb/volumes" Oct 30 05:20:11.213439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:11.213423 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a85529-1cd0-42f4-be40-edefe3420042" path="/var/lib/kubelet/pods/88a85529-1cd0-42f4-be40-edefe3420042/volumes" Oct 30 05:20:12.645881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:12.645847 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:20:12.646392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:12.646064 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" podUID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" containerName="maas-api" containerID="cri-o://a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69" gracePeriod=30 Oct 30 05:20:13.080886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.080856 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:20:13.228284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.207000 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") pod \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.254531 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls" pod "d91a2f1f-38a4-4d93-b061-86fb9be22e38" (UID: "d91a2f1f-38a4-4d93-b061-86fb9be22e38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.254686 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\" (UniqueName: \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\") pod \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.254801 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\") pod \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.254859 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\") pod \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\" (UID: \"d91a2f1f-38a4-4d93-b061-86fb9be22e38\") " Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.255616 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:13.268544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.267386 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs" pod "d91a2f1f-38a4-4d93-b061-86fb9be22e38" (UID: "d91a2f1f-38a4-4d93-b061-86fb9be22e38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:13.283992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.282191 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca" pod "d91a2f1f-38a4-4d93-b061-86fb9be22e38" (UID: "d91a2f1f-38a4-4d93-b061-86fb9be22e38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:13.291043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.286442 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h" pod "d91a2f1f-38a4-4d93-b061-86fb9be22e38" (UID: "d91a2f1f-38a4-4d93-b061-86fb9be22e38"). InnerVolumeSpecName "kube-api-access-sc87h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:13.350950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.350788 5058 generic.go:356] "Generic (PLEG): container finished" podID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" containerID="a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69" exitCode=0 Oct 30 05:20:13.350950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.350940 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" Oct 30 05:20:13.351171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.351050 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" event={"ID":"d91a2f1f-38a4-4d93-b061-86fb9be22e38","Type":"ContainerDied","Data":"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69"} Oct 30 05:20:13.351171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.351109 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp" event={"ID":"d91a2f1f-38a4-4d93-b061-86fb9be22e38","Type":"ContainerDied","Data":"265d97d1bc3d7aeb036d5739018387542239943b9d2c13febec916a2a333ed6d"} Oct 30 05:20:13.351171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.351139 5058 scope.go:122] "RemoveContainer" containerID="a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69" Oct 30 05:20:13.358412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.357970 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc87h\" (UniqueName: \"kubernetes.io/projected/d91a2f1f-38a4-4d93-b061-86fb9be22e38-kube-api-access-sc87h\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:13.358412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.357998 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91a2f1f-38a4-4d93-b061-86fb9be22e38-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:13.358412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.358387 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91a2f1f-38a4-4d93-b061-86fb9be22e38-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:13.366297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.366278 5058 scope.go:122] "RemoveContainer" containerID="a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69" Oct 30 05:20:13.366680 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:13.366657 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69\": container with ID starting with a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69 not found: ID does not exist" containerID="a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69" Oct 30 05:20:13.366800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.366777 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69"} err="failed to get container status \"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69\": rpc error: code = NotFound desc = could not find container \"a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69\": container with ID starting with a76cfd4667c6eba7f11facbeeaa8c8489fd4418414ba5a17789b1a3934317b69 not found: ID does not exist" Oct 30 05:20:13.408949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.408920 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:20:13.420927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:13.420898 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-010013-74769ff8bb-8mhvp"] Oct 30 05:20:15.164937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:15.164905 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" path="/var/lib/kubelet/pods/d91a2f1f-38a4-4d93-b061-86fb9be22e38/volumes" Oct 30 05:20:18.649964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:18.649930 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:20:19.193144 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:19.193106 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:20:23.655521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:23.655468 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" containerID="cri-o://8c812df0c4a91df339b491fb24768d9fcb674fc4daeb92c24bcc130f1a84fa6a" gracePeriod=25 Oct 30 05:20:29.320907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:29.320868 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:38.657518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.657481 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:20:38.658285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658269 5058 cpu_manager.go:414] "containerMap: removing container" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" Oct 30 05:20:38.658342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658299 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" Oct 30 05:20:38.658342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658315 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" Oct 30 05:20:38.658342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658321 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658349 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658354 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658372 5058 cpu_manager.go:414] "containerMap: removing container" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerName="maas-api" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658377 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerName="maas-api" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658391 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" containerName="maas-api" Oct 30 05:20:38.658461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658395 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" containerName="maas-api" Oct 30 05:20:38.658631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658507 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dbdf1d-648c-4b6e-a896-acddc715558c" containerName="maas-api" Oct 30 05:20:38.658631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658516 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91a2f1f-38a4-4d93-b061-86fb9be22e38" containerName="maas-api" Oct 30 05:20:38.658631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658527 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a85529-1cd0-42f4-be40-edefe3420042" containerName="payload-pre-processing" Oct 30 05:20:38.658631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658535 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2fb18e-b987-4fe5-8e93-845d5d0a50c3" containerName="payload-pre-processing" Oct 30 05:20:38.658631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.658545 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0a6347-86f9-4734-adb7-71037d2b38cb" containerName="payload-processing" Oct 30 05:20:38.663805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.663784 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.682877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.682845 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:20:38.690813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.690777 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a8f18-openshift-default-dockercfg-c6srh\"" Oct 30 05:20:38.749682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.749651 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.749827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.749699 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.749883 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.749846 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.749920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.749903 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.750034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.750013 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.750092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.750058 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nph\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.750137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.750125 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.750173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.750156 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.750354 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.750340 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852051 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852114 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nph\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852157 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852198 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852362 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852460 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852497 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852517 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852604 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852636 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852702 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852757 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852874 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.852993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.852973 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.854760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.854734 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.855147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.855131 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.873024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.873001 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.873330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.873308 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nph\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:38.978849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:38.978769 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:39.302168 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:20:39.302140 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8463a0_07eb_4ece_aaee_6867150eafca.slice/crio-5a4a8fdb5aae0b66da13409d9b15672b8c7d409fb6688b4d9c32ac1704d33da2 WatchSource:0}: Error finding container 5a4a8fdb5aae0b66da13409d9b15672b8c7d409fb6688b4d9c32ac1704d33da2: Status 404 returned error can't find the container with id 5a4a8fdb5aae0b66da13409d9b15672b8c7d409fb6688b4d9c32ac1704d33da2 Oct 30 05:20:39.304265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.304242 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:20:39.305365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.305338 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:39.305424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.305399 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:39.305459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.305427 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:39.321093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.321044 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:39.737008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.736957 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" event={"ID":"3b8463a0-07eb-4ece-aaee-6867150eafca","Type":"ContainerStarted","Data":"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2"} Oct 30 05:20:39.737515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.737048 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" event={"ID":"3b8463a0-07eb-4ece-aaee-6867150eafca","Type":"ContainerStarted","Data":"5a4a8fdb5aae0b66da13409d9b15672b8c7d409fb6688b4d9c32ac1704d33da2"} Oct 30 05:20:39.783459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.783404 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podStartSLOduration=1.783390261 podStartE2EDuration="1.783390261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:39.780920638 +0000 UTC m=+2212.750415627" watchObservedRunningTime="2026-10-30 05:20:39.783390261 +0000 UTC m=+2212.752885248" Oct 30 05:20:39.979517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:39.979484 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:40.980170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:40.980140 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 05:20:40.980652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:40.980193 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 05:20:41.985239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:41.979509 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 05:20:41.985239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:41.979582 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 05:20:42.979286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:42.979246 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 05:20:42.979490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:42.979301 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 05:20:43.884075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:43.883760 5058 generic.go:356] "Generic (PLEG): container finished" podID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerID="8c812df0c4a91df339b491fb24768d9fcb674fc4daeb92c24bcc130f1a84fa6a" exitCode=0 Oct 30 05:20:43.884075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:43.883992 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" event={"ID":"9bf5cd1c-b395-4027-8cba-24c083467d2b","Type":"ContainerDied","Data":"8c812df0c4a91df339b491fb24768d9fcb674fc4daeb92c24bcc130f1a84fa6a"} Oct 30 05:20:43.979611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:43.979570 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 05:20:43.979799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:43.979647 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 05:20:44.095699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.095672 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:20:44.216370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.216290 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\") pod \"9bf5cd1c-b395-4027-8cba-24c083467d2b\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " Oct 30 05:20:44.216370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.216355 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\" (UniqueName: \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\") pod \"9bf5cd1c-b395-4027-8cba-24c083467d2b\" (UID: \"9bf5cd1c-b395-4027-8cba-24c083467d2b\") " Oct 30 05:20:44.216728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.216706 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume" pod "9bf5cd1c-b395-4027-8cba-24c083467d2b" (UID: "9bf5cd1c-b395-4027-8cba-24c083467d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:44.217047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.217031 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf5cd1c-b395-4027-8cba-24c083467d2b-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:44.218611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.218579 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r" pod "9bf5cd1c-b395-4027-8cba-24c083467d2b" (UID: "9bf5cd1c-b395-4027-8cba-24c083467d2b"). InnerVolumeSpecName "kube-api-access-7b25r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:44.318679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.318650 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b25r\" (UniqueName: \"kubernetes.io/projected/9bf5cd1c-b395-4027-8cba-24c083467d2b-kube-api-access-7b25r\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:44.892465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.892436 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" Oct 30 05:20:44.892886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.892467 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk" event={"ID":"9bf5cd1c-b395-4027-8cba-24c083467d2b","Type":"ContainerDied","Data":"00b317e4b01dc1b8540420dec3b9b2371018712477e69e294d4800dcbcd6dc59"} Oct 30 05:20:44.892886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.892535 5058 scope.go:122] "RemoveContainer" containerID="8c812df0c4a91df339b491fb24768d9fcb674fc4daeb92c24bcc130f1a84fa6a" Oct 30 05:20:44.960653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.960624 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:20:44.973955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.973924 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk"] Oct 30 05:20:44.979890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.979866 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 05:20:44.980012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:44.979911 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 05:20:45.165081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:45.165003 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" path="/var/lib/kubelet/pods/9bf5cd1c-b395-4027-8cba-24c083467d2b/volumes" Oct 30 05:20:46.136069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:46.136035 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:46.137259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:46.137200 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:46.138351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:46.138331 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:20:46.216967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:46.216936 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 05:20:46.217397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:46.217368 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" containerID="cri-o://a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516" gracePeriod=30 Oct 30 05:20:49.279883 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:49.279832 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:20:51.501027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.501003 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 05:20:51.592247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592199 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592620 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592665 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592727 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592776 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592818 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.592911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592850 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.593203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.592933 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:51.593203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593017 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.593203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593174 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\") pod \"2230217f-bdf0-48a5-a885-14836a50bc22\" (UID: \"2230217f-bdf0-48a5-a885-14836a50bc22\") " Oct 30 05:20:51.593638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593611 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:51.593750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593651 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:51.593965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593940 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:51.594065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.593962 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:51.594169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594146 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.594273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594175 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230217f-bdf0-48a5-a885-14836a50bc22-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.594273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594190 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.594273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594203 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.594273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594230 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.594895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.594870 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:51.595940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.595917 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:51.595940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.595920 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:20:51.596551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.596528 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp" pod "2230217f-bdf0-48a5-a885-14836a50bc22" (UID: "2230217f-bdf0-48a5-a885-14836a50bc22"). InnerVolumeSpecName "kube-api-access-bhgkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:51.695822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.695793 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgkp\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-kube-api-access-bhgkp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.695822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.695820 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230217f-bdf0-48a5-a885-14836a50bc22-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.695822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.695830 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230217f-bdf0-48a5-a885-14836a50bc22-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.696056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.695838 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230217f-bdf0-48a5-a885-14836a50bc22-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.987398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.987355 5058 generic.go:356] "Generic (PLEG): container finished" podID="2230217f-bdf0-48a5-a885-14836a50bc22" containerID="a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516" exitCode=0 Oct 30 05:20:51.987398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.987400 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" Oct 30 05:20:51.987603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.987433 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" event={"ID":"2230217f-bdf0-48a5-a885-14836a50bc22","Type":"ContainerDied","Data":"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516"} Oct 30 05:20:51.987603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.987486 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw" event={"ID":"2230217f-bdf0-48a5-a885-14836a50bc22","Type":"ContainerDied","Data":"6427010f444aa6d703fa7a13baa08a4d3690b009b6fd3002eb88ae4d85a3841e"} Oct 30 05:20:51.987603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.987516 5058 scope.go:122] "RemoveContainer" containerID="a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516" Oct 30 05:20:51.999513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.999487 5058 scope.go:122] "RemoveContainer" containerID="a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516" Oct 30 05:20:51.999810 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:20:51.999789 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516\": container with ID starting with a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516 not found: ID does not exist" containerID="a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516" Oct 30 05:20:51.999903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:51.999824 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516"} err="failed to get container status \"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516\": rpc error: code = NotFound desc = could not find container \"a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516\": container with ID starting with a3d3bc96f9fb35e230f2c14534aaff42b656905ff79ddbf6bcb5c0ab6a1b6516 not found: ID does not exist" Oct 30 05:20:52.058412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:52.058378 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 05:20:52.071650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:52.071624 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw"] Oct 30 05:20:53.164195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:53.164161 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" path="/var/lib/kubelet/pods/2230217f-bdf0-48a5-a885-14836a50bc22/volumes" Oct 30 05:20:55.663977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:55.663943 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:20:55.664505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:55.664162 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" podUID="f59235d6-516a-435b-93f8-16dbc4583ce7" containerName="maas-api" containerID="cri-o://589a3ca1466d3ac2d52cd798b483e443d8f127a8a095f55b47aa86deef8ec5ec" gracePeriod=30 Oct 30 05:20:56.046985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.046787 5058 generic.go:356] "Generic (PLEG): container finished" podID="f59235d6-516a-435b-93f8-16dbc4583ce7" containerID="589a3ca1466d3ac2d52cd798b483e443d8f127a8a095f55b47aa86deef8ec5ec" exitCode=0 Oct 30 05:20:56.046985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.046853 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" event={"ID":"f59235d6-516a-435b-93f8-16dbc4583ce7","Type":"ContainerDied","Data":"589a3ca1466d3ac2d52cd798b483e443d8f127a8a095f55b47aa86deef8ec5ec"} Oct 30 05:20:56.217452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.217418 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:20:56.397025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.396985 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\" (UniqueName: \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\") pod \"f59235d6-516a-435b-93f8-16dbc4583ce7\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " Oct 30 05:20:56.397305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.397070 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\") pod \"f59235d6-516a-435b-93f8-16dbc4583ce7\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " Oct 30 05:20:56.397305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.397119 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\") pod \"f59235d6-516a-435b-93f8-16dbc4583ce7\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " Oct 30 05:20:56.397305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.397179 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\") pod \"f59235d6-516a-435b-93f8-16dbc4583ce7\" (UID: \"f59235d6-516a-435b-93f8-16dbc4583ce7\") " Oct 30 05:20:56.398544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.398508 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca" pod "f59235d6-516a-435b-93f8-16dbc4583ce7" (UID: "f59235d6-516a-435b-93f8-16dbc4583ce7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:56.404556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.404531 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs" pod "f59235d6-516a-435b-93f8-16dbc4583ce7" (UID: "f59235d6-516a-435b-93f8-16dbc4583ce7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:56.406683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.406654 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9" pod "f59235d6-516a-435b-93f8-16dbc4583ce7" (UID: "f59235d6-516a-435b-93f8-16dbc4583ce7"). InnerVolumeSpecName "kube-api-access-d94p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:56.406797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.406722 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls" pod "f59235d6-516a-435b-93f8-16dbc4583ce7" (UID: "f59235d6-516a-435b-93f8-16dbc4583ce7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:56.502254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.500413 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94p9\" (UniqueName: \"kubernetes.io/projected/f59235d6-516a-435b-93f8-16dbc4583ce7-kube-api-access-d94p9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:56.502254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.500453 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:56.502254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.500468 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59235d6-516a-435b-93f8-16dbc4583ce7-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:56.502254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:56.500480 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59235d6-516a-435b-93f8-16dbc4583ce7-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:20:57.054920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.054882 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" event={"ID":"f59235d6-516a-435b-93f8-16dbc4583ce7","Type":"ContainerDied","Data":"4bb55e38c75286aa1437ce51ef298620d730b14cb92d311217be40f782b5b280"} Oct 30 05:20:57.055357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.054932 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl" Oct 30 05:20:57.055357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.054981 5058 scope.go:122] "RemoveContainer" containerID="589a3ca1466d3ac2d52cd798b483e443d8f127a8a095f55b47aa86deef8ec5ec" Oct 30 05:20:57.119804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.119771 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:20:57.126796 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.126771 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7f962-58bc78b586-mhfkl"] Oct 30 05:20:57.164119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:57.164088 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59235d6-516a-435b-93f8-16dbc4583ce7" path="/var/lib/kubelet/pods/f59235d6-516a-435b-93f8-16dbc4583ce7/volumes" Oct 30 05:20:58.739266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:20:58.737900 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:21:03.537601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:03.537562 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:21:03.538081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:03.537828 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" containerID="cri-o://54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9" gracePeriod=30 Oct 30 05:21:08.843004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.842976 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:21:08.846198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846173 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846317 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846353 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846483 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846522 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846554 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846572 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:08.846685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846645 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846745 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.846854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846830 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:08.846905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846857 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:08.846952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.846901 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\") pod \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\" (UID: \"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a\") " Oct 30 05:21:08.847103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847071 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:08.847103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847072 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:08.847678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847657 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.847737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847685 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.847737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847704 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.847737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847719 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.847737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.847732 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.848740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.848712 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:08.848843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.848759 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "kube-api-access-qtpkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:08.848843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.848779 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:08.848924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.848870 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token" pod "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" (UID: "f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:08.948846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.948818 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.948846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.948843 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.948846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.948852 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:08.949134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:08.948860 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtpkh\" (UniqueName: \"kubernetes.io/projected/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a-kube-api-access-qtpkh\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:09.163120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.163080 5058 generic.go:356] "Generic (PLEG): container finished" podID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerID="54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9" exitCode=0 Oct 30 05:21:09.163327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.163185 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" Oct 30 05:21:09.165480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.165458 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerDied","Data":"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9"} Oct 30 05:21:09.165585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.165509 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs" event={"ID":"f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a","Type":"ContainerDied","Data":"20bc767b4ce891efd3b9ab8640f46d472433b83e72c1fc80a27297070acdff0f"} Oct 30 05:21:09.165585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.165533 5058 scope.go:122] "RemoveContainer" containerID="54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9" Oct 30 05:21:09.177164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.177147 5058 scope.go:122] "RemoveContainer" containerID="bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8" Oct 30 05:21:09.187186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.187170 5058 scope.go:122] "RemoveContainer" containerID="54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9" Oct 30 05:21:09.187449 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:09.187432 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9\": container with ID starting with 54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9 not found: ID does not exist" containerID="54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9" Oct 30 05:21:09.187523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.187458 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9"} err="failed to get container status \"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9\": rpc error: code = NotFound desc = could not find container \"54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9\": container with ID starting with 54222f8e28741357a7a8b4c6c447131e7233cdad27c8b2b22ae9a7d8eb02daf9 not found: ID does not exist" Oct 30 05:21:09.187523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.187474 5058 scope.go:122] "RemoveContainer" containerID="bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8" Oct 30 05:21:09.187708 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:09.187685 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8\": container with ID starting with bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8 not found: ID does not exist" containerID="bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8" Oct 30 05:21:09.187751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.187716 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8"} err="failed to get container status \"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8\": rpc error: code = NotFound desc = could not find container \"bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8\": container with ID starting with bc1d0df534f2816ce508ef71d9838a477d051ae9a3a283ad6b51d5885ef91df8 not found: ID does not exist" Oct 30 05:21:09.221008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.220989 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:21:09.224682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:09.224659 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs"] Oct 30 05:21:11.163111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:11.163073 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" path="/var/lib/kubelet/pods/f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a/volumes" Oct 30 05:21:13.762231 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:13.762179 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" podUID="ef7cd480-6be8-4564-9f8e-86bc3592133f" containerName="main" containerID="cri-o://ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d" gracePeriod=45 Oct 30 05:21:14.058542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.058507 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:21:14.133538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133506 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.133730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133581 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.133730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133627 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.133730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133697 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\" (UniqueName: \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.133909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133804 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:14.133909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133860 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.134018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.133996 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\") pod \"ef7cd480-6be8-4564-9f8e-86bc3592133f\" (UID: \"ef7cd480-6be8-4564-9f8e-86bc3592133f\") " Oct 30 05:21:14.134382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.134281 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:14.134525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.134479 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:14.134869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.134841 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.134986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.134870 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.134986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.134884 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.136517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.136484 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:14.136620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.136581 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "kube-api-access-cfgbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:14.136685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.136520 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs" pod "ef7cd480-6be8-4564-9f8e-86bc3592133f" (UID: "ef7cd480-6be8-4564-9f8e-86bc3592133f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:14.201046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.201011 5058 generic.go:356] "Generic (PLEG): container finished" podID="ef7cd480-6be8-4564-9f8e-86bc3592133f" containerID="ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d" exitCode=0 Oct 30 05:21:14.201214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.201055 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" Oct 30 05:21:14.201214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.201087 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" event={"ID":"ef7cd480-6be8-4564-9f8e-86bc3592133f","Type":"ContainerDied","Data":"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d"} Oct 30 05:21:14.201214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.201125 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql" event={"ID":"ef7cd480-6be8-4564-9f8e-86bc3592133f","Type":"ContainerDied","Data":"8e0e03076efa79f3eebd869887272ebb7095fdbe0ad798b58529a87893fb3612"} Oct 30 05:21:14.201214 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.201143 5058 scope.go:122] "RemoveContainer" containerID="ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d" Oct 30 05:21:14.212854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.212831 5058 scope.go:122] "RemoveContainer" containerID="ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d" Oct 30 05:21:14.213122 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:14.213100 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d\": container with ID starting with ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d not found: ID does not exist" containerID="ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d" Oct 30 05:21:14.213198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.213129 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d"} err="failed to get container status \"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d\": rpc error: code = NotFound desc = could not find container \"ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d\": container with ID starting with ff9889e95ab0bbe50e458ac5de1ad6254f20be984269bbbd242c10fac50b488d not found: ID does not exist" Oct 30 05:21:14.236429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.236406 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7cd480-6be8-4564-9f8e-86bc3592133f-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.236429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.236426 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgbx\" (UniqueName: \"kubernetes.io/projected/ef7cd480-6be8-4564-9f8e-86bc3592133f-kube-api-access-cfgbx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.236429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.236436 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7cd480-6be8-4564-9f8e-86bc3592133f-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:14.267315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.267285 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:21:14.274138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.274078 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ccafe0/test-model-ccafe0-kserve-85d5b9ffc9-c6rql"] Oct 30 05:21:14.809741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.809708 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:21:14.810246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:14.809963 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" containerID="cri-o://2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16" gracePeriod=30 Oct 30 05:21:15.164585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:15.164545 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7cd480-6be8-4564-9f8e-86bc3592133f" path="/var/lib/kubelet/pods/ef7cd480-6be8-4564-9f8e-86bc3592133f/volumes" Oct 30 05:21:19.198009 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:19.197973 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:21:20.189950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.189901 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:21:20.289987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.289952 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290011 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290070 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290116 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290258 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290361 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290478 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290536 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290599 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290627 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\") pod \"13b0bc50-eba6-468e-a81e-1deb86c601ab\" (UID: \"13b0bc50-eba6-468e-a81e-1deb86c601ab\") " Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290729 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.290777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290754 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:20.291086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.290926 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.291086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291010 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.291516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291492 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.291629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291519 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b0bc50-eba6-468e-a81e-1deb86c601ab-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.291629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291532 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.291629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291545 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.291629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.291557 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.293049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.293011 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:20.293544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.293518 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "kube-api-access-glgj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:20.293695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.293605 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.293745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.293697 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token" pod "13b0bc50-eba6-468e-a81e-1deb86c601ab" (UID: "13b0bc50-eba6-468e-a81e-1deb86c601ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:20.386270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.386167 5058 generic.go:356] "Generic (PLEG): container finished" podID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerID="2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16" exitCode=0 Oct 30 05:21:20.386421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.386269 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" Oct 30 05:21:20.386461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.386430 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" event={"ID":"13b0bc50-eba6-468e-a81e-1deb86c601ab","Type":"ContainerDied","Data":"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16"} Oct 30 05:21:20.386525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.386511 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764" event={"ID":"13b0bc50-eba6-468e-a81e-1deb86c601ab","Type":"ContainerDied","Data":"2abbd1faac12edca833022c02551bfa7b8964ca331e488209b946eae92622f8f"} Oct 30 05:21:20.386565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.386544 5058 scope.go:122] "RemoveContainer" containerID="2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16" Oct 30 05:21:20.392848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.392823 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.392932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.392855 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glgj6\" (UniqueName: \"kubernetes.io/projected/13b0bc50-eba6-468e-a81e-1deb86c601ab-kube-api-access-glgj6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.392932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.392871 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.392932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.392885 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b0bc50-eba6-468e-a81e-1deb86c601ab-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.400560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.400543 5058 scope.go:122] "RemoveContainer" containerID="2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16" Oct 30 05:21:20.400826 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:20.400808 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16\": container with ID starting with 2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16 not found: ID does not exist" containerID="2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16" Oct 30 05:21:20.400880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.400833 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16"} err="failed to get container status \"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16\": rpc error: code = NotFound desc = could not find container \"2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16\": container with ID starting with 2ea98ff8093f5f30f96aa029fc80fb9cff676a088d98b04d19ac8bfe1a8b1f16 not found: ID does not exist" Oct 30 05:21:20.457468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.457435 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:21:20.461004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:20.460980 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764"] Oct 30 05:21:21.170195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:21.170146 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" path="/var/lib/kubelet/pods/13b0bc50-eba6-468e-a81e-1deb86c601ab/volumes" Oct 30 05:21:26.285648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:26.285611 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 05:21:26.286141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:26.285852 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" containerID="cri-o://cd245ac0f3b0b18a5cf7d4fd5461060509419a970864ddba5b87803b57b8d558" gracePeriod=30 Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.464916 5058 generic.go:356] "Generic (PLEG): container finished" podID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerID="637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e" exitCode=0 Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.465164 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerDied","Data":"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e"} Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.466200 5058 scope.go:122] "RemoveContainer" containerID="637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e" Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.468832 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.468939 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:27.476140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:27.468990 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:28.475726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:28.475694 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerStarted","Data":"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9"} Oct 30 05:21:28.589742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:28.589698 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:21:28.590645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:28.590519 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:28.590857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:28.590833 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:29.275260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:29.275205 5058 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:21:29.275429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:29.275292 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:21:29.590183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:29.590075 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:29.590183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:29.590153 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:30.597251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:30.595339 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:30.597251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:30.595401 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:31.374263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.358195 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 05:21:31.374263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.373894 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" containerID="cri-o://165e149deb2f6722036de332086b502cbee9145da7d541b1dcbcb7c761e34a9a" gracePeriod=30 Oct 30 05:21:31.510164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.510123 5058 generic.go:356] "Generic (PLEG): container finished" podID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerID="cd245ac0f3b0b18a5cf7d4fd5461060509419a970864ddba5b87803b57b8d558" exitCode=0 Oct 30 05:21:31.510486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.510393 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" event={"ID":"3458257c-f753-4c33-8745-496d7f9ff5d9","Type":"ContainerDied","Data":"cd245ac0f3b0b18a5cf7d4fd5461060509419a970864ddba5b87803b57b8d558"} Oct 30 05:21:31.587229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.587130 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:31.587396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.587191 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:31.663269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.663246 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 05:21:31.789298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789260 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789348 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789479 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789637 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789714 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789763 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789836 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.789900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789870 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:31.789952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789912 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.790001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.789991 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\") pod \"3458257c-f753-4c33-8745-496d7f9ff5d9\" (UID: \"3458257c-f753-4c33-8745-496d7f9ff5d9\") " Oct 30 05:21:31.790872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.790679 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:31.790872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.790855 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.791235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.790878 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3458257c-f753-4c33-8745-496d7f9ff5d9-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.791317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.791294 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:31.791384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.791352 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:31.791451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.791401 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:31.792324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.792289 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:31.793099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.793074 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "kube-api-access-7cqw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:31.793559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.793535 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:31.794984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.794960 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo" pod "3458257c-f753-4c33-8745-496d7f9ff5d9" (UID: "3458257c-f753-4c33-8745-496d7f9ff5d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893169 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893204 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893245 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cqw4\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-kube-api-access-7cqw4\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893261 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893279 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893309 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:31.895242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:31.893325 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3458257c-f753-4c33-8745-496d7f9ff5d9-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:32.519795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.519750 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" event={"ID":"3458257c-f753-4c33-8745-496d7f9ff5d9","Type":"ContainerDied","Data":"89cccc068a892ccfa1f856fb5f214859be82708ef2e5b2d3975c98ead45a539d"} Oct 30 05:21:32.519795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.519789 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5" Oct 30 05:21:32.520033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.519823 5058 scope.go:122] "RemoveContainer" containerID="cd245ac0f3b0b18a5cf7d4fd5461060509419a970864ddba5b87803b57b8d558" Oct 30 05:21:32.586941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.586902 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:32.587135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.586960 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:32.592592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.592565 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 05:21:32.605552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:32.605523 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5"] Oct 30 05:21:33.163210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:33.163179 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" path="/var/lib/kubelet/pods/3458257c-f753-4c33-8745-496d7f9ff5d9/volumes" Oct 30 05:21:33.594098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:33.593384 5058 patch_prober.go:29] interesting pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:21:33.594098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:33.593461 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 05:21:34.419333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:34.419298 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:21:34.592195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:34.592155 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:21:34.593556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:34.593535 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:21:34.741457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:34.741360 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:21:34.741457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:34.741428 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:21:36.160277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.160240 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:21:36.160973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.160959 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.160975 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.160984 5058 cpu_manager.go:414] "containerMap: removing container" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.160989 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161008 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161015 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" Oct 30 05:21:36.161023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161024 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161029 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161043 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ef7cd480-6be8-4564-9f8e-86bc3592133f" containerName="main" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161047 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7cd480-6be8-4564-9f8e-86bc3592133f" containerName="main" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161057 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161061 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161071 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161075 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161087 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f59235d6-516a-435b-93f8-16dbc4583ce7" containerName="maas-api" Oct 30 05:21:36.161202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161091 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59235d6-516a-435b-93f8-16dbc4583ce7" containerName="maas-api" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161214 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3458257c-f753-4c33-8745-496d7f9ff5d9" containerName="istio-proxy" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161236 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b0bc50-eba6-468e-a81e-1deb86c601ab" containerName="istio-proxy" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161247 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf5cd1c-b395-4027-8cba-24c083467d2b" containerName="payload-processing" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161254 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161261 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7cd480-6be8-4564-9f8e-86bc3592133f" containerName="main" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161268 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97c5ba4-a3f6-47c6-8cc5-0442c2ae3e2a" containerName="istio-proxy" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161275 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2230217f-bdf0-48a5-a885-14836a50bc22" containerName="istio-proxy" Oct 30 05:21:36.161520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.161283 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59235d6-516a-435b-93f8-16dbc4583ce7" containerName="maas-api" Oct 30 05:21:36.166724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.166702 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.184853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.184832 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:21:36.185712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.185690 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0dba31\"" Oct 30 05:21:36.259510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.259469 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65k2\" (UniqueName: \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.259676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.259619 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.368241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.365532 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65k2\" (UniqueName: \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.368241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.365689 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.368241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.366632 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.472459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.467250 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65k2\" (UniqueName: \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\") pod \"payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.482983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.482952 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:36.615762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.612648 5058 generic.go:356] "Generic (PLEG): container finished" podID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerID="165e149deb2f6722036de332086b502cbee9145da7d541b1dcbcb7c761e34a9a" exitCode=0 Oct 30 05:21:36.615762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.612889 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" event={"ID":"31475025-84fc-4d3f-acc3-2241c6eaa032","Type":"ContainerDied","Data":"165e149deb2f6722036de332086b502cbee9145da7d541b1dcbcb7c761e34a9a"} Oct 30 05:21:36.832977 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:21:36.832947 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcdd147_8a32_4b22_8538_7e9ee4249c8c.slice/crio-c64e7a311ab951871845f2ec992ca249a829a104c77c1c3c5d03e2bfda26b8a7 WatchSource:0}: Error finding container c64e7a311ab951871845f2ec992ca249a829a104c77c1c3c5d03e2bfda26b8a7: Status 404 returned error can't find the container with id c64e7a311ab951871845f2ec992ca249a829a104c77c1c3c5d03e2bfda26b8a7 Oct 30 05:21:36.835803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:36.835778 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:21:37.080127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.080067 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185365 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185463 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185593 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185655 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185726 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185793 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185869 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.186301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.185901 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.187723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.187440 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:37.188822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.188802 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\") pod \"31475025-84fc-4d3f-acc3-2241c6eaa032\" (UID: \"31475025-84fc-4d3f-acc3-2241c6eaa032\") " Oct 30 05:21:37.189674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.189646 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.190772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.189856 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:37.190925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.190002 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:37.191022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.190267 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:37.191100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.190731 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:37.192010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.191982 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:37.192284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.192211 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:37.192699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.192675 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "kube-api-access-kgdgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:37.192781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.192716 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token" pod "31475025-84fc-4d3f-acc3-2241c6eaa032" (UID: "31475025-84fc-4d3f-acc3-2241c6eaa032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:37.292609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292579 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292606 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31475025-84fc-4d3f-acc3-2241c6eaa032-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292617 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292626 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292633 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292641 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdgb\" (UniqueName: \"kubernetes.io/projected/31475025-84fc-4d3f-acc3-2241c6eaa032-kube-api-access-kgdgb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292649 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.292846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.292659 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31475025-84fc-4d3f-acc3-2241c6eaa032-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:37.622362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.622327 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" Oct 30 05:21:37.622542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.622361 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql" event={"ID":"31475025-84fc-4d3f-acc3-2241c6eaa032","Type":"ContainerDied","Data":"72ed093fb9e6d93d2cf1c62e79a80de2eadcdd3a47b315834e69dc38e8c77628"} Oct 30 05:21:37.622542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.622437 5058 scope.go:122] "RemoveContainer" containerID="165e149deb2f6722036de332086b502cbee9145da7d541b1dcbcb7c761e34a9a" Oct 30 05:21:37.624292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.624262 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" event={"ID":"bdcdd147-8a32-4b22-8538-7e9ee4249c8c","Type":"ContainerStarted","Data":"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011"} Oct 30 05:21:37.624406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.624317 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" event={"ID":"bdcdd147-8a32-4b22-8538-7e9ee4249c8c","Type":"ContainerStarted","Data":"c64e7a311ab951871845f2ec992ca249a829a104c77c1c3c5d03e2bfda26b8a7"} Oct 30 05:21:37.624406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.624350 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:37.678824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.678732 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" podStartSLOduration=1.678714819 podStartE2EDuration="1.678714819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:37.678413705 +0000 UTC m=+2270.647908704" watchObservedRunningTime="2026-10-30 05:21:37.678714819 +0000 UTC m=+2270.648209811" Oct 30 05:21:37.725628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.725552 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 05:21:37.732955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:37.732924 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql"] Oct 30 05:21:39.163358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:39.163319 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" path="/var/lib/kubelet/pods/31475025-84fc-4d3f-acc3-2241c6eaa032/volumes" Oct 30 05:21:39.496136 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:39.496055 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:21:39.522859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:39.522822 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:21:43.694331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.694299 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:21:43.695298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.695275 5058 cpu_manager.go:414] "containerMap: removing container" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" Oct 30 05:21:43.695298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.695296 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" Oct 30 05:21:43.695542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.695527 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="31475025-84fc-4d3f-acc3-2241c6eaa032" containerName="istio-proxy" Oct 30 05:21:43.699315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.699289 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.719412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.719386 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:21:43.725726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.725691 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.725862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.725819 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.725862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.725854 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.725971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.725946 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfgc\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.726029 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.725996 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.726088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.726054 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.726142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.726093 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.726142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.726132 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.726253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.726152 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.727757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.727737 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-68b893-openshift-default-dockercfg-6ns6m\"" Oct 30 05:21:43.827213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827174 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfgc\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827257 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827302 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827332 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827358 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827372 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827423 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827488 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827509 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827806 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.827892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.827834 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.828104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.828080 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.828174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.828086 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.828277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.828252 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.829911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.829885 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.829911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.829902 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.847612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.847588 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfgc\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.847732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.847587 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:43.967811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.967732 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:21:43.967975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:43.967938 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" podUID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" containerName="maas-api" containerID="cri-o://9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273" gracePeriod=30 Oct 30 05:21:44.016076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.016047 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:44.389292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.386867 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:21:44.396582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.396538 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:21:44.409043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.409008 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:44.409291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.409267 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:44.409471 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.409445 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:44.483697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.481489 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\") pod \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " Oct 30 05:21:44.483697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.481552 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\") pod \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " Oct 30 05:21:44.483697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.481646 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") pod \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " Oct 30 05:21:44.483697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.481724 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\" (UniqueName: \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\") pod \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\" (UID: \"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef\") " Oct 30 05:21:44.484852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.484531 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca" pod "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" (UID: "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:44.500131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.499368 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" podUID="d4047510-acbb-41cf-aa18-3817bd67037c" containerName="payload-processing" containerID="cri-o://bb40b0bedfac54a894442ad67509cd0ddf9f35d0373940fbfb9e2fc11120b2e8" gracePeriod=25 Oct 30 05:21:44.530393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.525112 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" containerID="cri-o://f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6" gracePeriod=25 Oct 30 05:21:44.585692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.585577 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.586857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.586800 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs" pod "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" (UID: "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:44.589083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.589021 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls" pod "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" (UID: "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:44.590603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.590560 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9" pod "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" (UID: "d3fe6ff0-d65c-4759-bf28-0f47cb9dddef"). InnerVolumeSpecName "kube-api-access-95gb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:44.685534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.684617 5058 generic.go:356] "Generic (PLEG): container finished" podID="d4047510-acbb-41cf-aa18-3817bd67037c" containerID="bb40b0bedfac54a894442ad67509cd0ddf9f35d0373940fbfb9e2fc11120b2e8" exitCode=0 Oct 30 05:21:44.685534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.684818 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" event={"ID":"d4047510-acbb-41cf-aa18-3817bd67037c","Type":"ContainerDied","Data":"bb40b0bedfac54a894442ad67509cd0ddf9f35d0373940fbfb9e2fc11120b2e8"} Oct 30 05:21:44.689083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.689050 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.689197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.689086 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.689197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.689100 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95gb9\" (UniqueName: \"kubernetes.io/projected/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef-kube-api-access-95gb9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.689772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.689746 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" event={"ID":"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f","Type":"ContainerStarted","Data":"412aacbd46ddea85dbfd42d5931b3333d1870e4a949b321b3af5c9f0faa4d27d"} Oct 30 05:21:44.691290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.691267 5058 generic.go:356] "Generic (PLEG): container finished" podID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" containerID="9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273" exitCode=0 Oct 30 05:21:44.691403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.691359 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" event={"ID":"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef","Type":"ContainerDied","Data":"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273"} Oct 30 05:21:44.691403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.691392 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" event={"ID":"d3fe6ff0-d65c-4759-bf28-0f47cb9dddef","Type":"ContainerDied","Data":"b3f02b40f8c8809834e5499a6856159bc616983918d20791473721b5282e2283"} Oct 30 05:21:44.691519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.691415 5058 scope.go:122] "RemoveContainer" containerID="9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273" Oct 30 05:21:44.691573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.691544 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99" Oct 30 05:21:44.706699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.706675 5058 scope.go:122] "RemoveContainer" containerID="9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273" Oct 30 05:21:44.707033 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:44.706969 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273\": container with ID starting with 9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273 not found: ID does not exist" containerID="9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273" Oct 30 05:21:44.707033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.707004 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273"} err="failed to get container status \"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273\": rpc error: code = NotFound desc = could not find container \"9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273\": container with ID starting with 9878549e423027d8af9454af71608c26e49a5690aeec46ed4045074a685ff273 not found: ID does not exist" Oct 30 05:21:44.761263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.759789 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:21:44.765985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.765296 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453511-d84f9cd79-8sl99"] Oct 30 05:21:44.895354 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:44.895321 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:21:45.102892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.102854 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\" (UniqueName: \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\") pod \"d4047510-acbb-41cf-aa18-3817bd67037c\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " Oct 30 05:21:45.103079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.102954 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\") pod \"d4047510-acbb-41cf-aa18-3817bd67037c\" (UID: \"d4047510-acbb-41cf-aa18-3817bd67037c\") " Oct 30 05:21:45.103382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.103357 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume" pod "d4047510-acbb-41cf-aa18-3817bd67037c" (UID: "d4047510-acbb-41cf-aa18-3817bd67037c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:45.103604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.103584 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4047510-acbb-41cf-aa18-3817bd67037c-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:45.105089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.105062 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq" pod "d4047510-acbb-41cf-aa18-3817bd67037c" (UID: "d4047510-acbb-41cf-aa18-3817bd67037c"). InnerVolumeSpecName "kube-api-access-zjbvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:45.164996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.164959 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" path="/var/lib/kubelet/pods/d3fe6ff0-d65c-4759-bf28-0f47cb9dddef/volumes" Oct 30 05:21:45.204808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.204782 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbvq\" (UniqueName: \"kubernetes.io/projected/d4047510-acbb-41cf-aa18-3817bd67037c-kube-api-access-zjbvq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:45.700339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.700298 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" event={"ID":"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f","Type":"ContainerStarted","Data":"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86"} Oct 30 05:21:45.702971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.702950 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" Oct 30 05:21:45.703098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.702984 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5" event={"ID":"d4047510-acbb-41cf-aa18-3817bd67037c","Type":"ContainerDied","Data":"f564fbbcaf1a3080fed8efe1be70fd23c5cdc4ad4c2ab5f1885f5b678fe76785"} Oct 30 05:21:45.703098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.703036 5058 scope.go:122] "RemoveContainer" containerID="bb40b0bedfac54a894442ad67509cd0ddf9f35d0373940fbfb9e2fc11120b2e8" Oct 30 05:21:45.755538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.755479 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podStartSLOduration=2.755464289 podStartE2EDuration="2.755464289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:45.749858293 +0000 UTC m=+2278.719353280" watchObservedRunningTime="2026-10-30 05:21:45.755464289 +0000 UTC m=+2278.724959276" Oct 30 05:21:45.790628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.790589 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:21:45.795466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:45.795432 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5"] Oct 30 05:21:46.017280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:46.017194 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:46.019487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:46.019464 5058 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 05:21:46.019602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:46.019508 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 05:21:47.017174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:47.017140 5058 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 05:21:47.017571 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:47.017203 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 05:21:47.184045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:47.184010 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4047510-acbb-41cf-aa18-3817bd67037c" path="/var/lib/kubelet/pods/d4047510-acbb-41cf-aa18-3817bd67037c/volumes" Oct 30 05:21:48.017075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.017043 5058 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 05:21:48.017276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.017103 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 05:21:48.418475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.418431 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:21:48.418697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.418654 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerName="maas-api" containerID="cri-o://6c0e6c8a7b8abc3d07a24ff5b43529053806bec508196412de40caaee0a485d5" gracePeriod=30 Oct 30 05:21:48.657802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.657763 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:21:48.902040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.893353 5058 generic.go:356] "Generic (PLEG): container finished" podID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerID="6c0e6c8a7b8abc3d07a24ff5b43529053806bec508196412de40caaee0a485d5" exitCode=0 Oct 30 05:21:48.902040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:48.893570 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" event={"ID":"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee","Type":"ContainerDied","Data":"6c0e6c8a7b8abc3d07a24ff5b43529053806bec508196412de40caaee0a485d5"} Oct 30 05:21:49.083072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.065888 5058 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 05:21:49.083072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.065956 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 05:21:49.202245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.200838 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:21:49.202245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.201403 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:49.300271 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:49.300237 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:21:49.309755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.309727 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\") pod \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " Oct 30 05:21:49.309881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.309840 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\" (UniqueName: \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\") pod \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " Oct 30 05:21:49.309881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.309865 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\") pod \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " Oct 30 05:21:49.310037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.310018 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\") pod \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\" (UID: \"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee\") " Oct 30 05:21:49.310612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.310575 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca" pod "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" (UID: "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:49.311541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.311519 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:49.313165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.313135 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs" pod "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" (UID: "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:49.313425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.313403 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls" pod "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" (UID: "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:49.313769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.313742 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5" pod "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" (UID: "ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee"). InnerVolumeSpecName "kube-api-access-wqwq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:49.412537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.412501 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:49.412537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.412534 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwq5\" (UniqueName: \"kubernetes.io/projected/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-kube-api-access-wqwq5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:49.412537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.412545 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:49.901741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.901710 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" Oct 30 05:21:49.901741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.901727 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" event={"ID":"ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee","Type":"ContainerDied","Data":"4443bb0774ddf22047720c6c140415c1a8c549885ec3852f198c03b169edc2f1"} Oct 30 05:21:49.901981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.901802 5058 scope.go:122] "RemoveContainer" containerID="6c0e6c8a7b8abc3d07a24ff5b43529053806bec508196412de40caaee0a485d5" Oct 30 05:21:49.973859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.973830 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:21:49.987230 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:49.987189 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp"] Oct 30 05:21:50.017041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:50.017011 5058 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 05:21:50.017186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:50.017066 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 05:21:51.164318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.164283 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" path="/var/lib/kubelet/pods/ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee/volumes" Oct 30 05:21:51.199797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.199760 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:51.200733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.200566 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:51.201469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.201448 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:21:51.829542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.829470 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccafe0-db577dc89-4mmxp" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.166:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:21:51.869123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.869088 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:21:51.869379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:51.869353 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" containerID="cri-o://fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d" gracePeriod=30 Oct 30 05:21:56.485905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:56.485865 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:21:56.486433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:56.486156 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" containerID="cri-o://d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef" gracePeriod=30 Oct 30 05:21:57.189181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.189153 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:21:57.234774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.234737 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.234893 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.234997 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235054 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235125 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235269 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235307 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235344 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235378 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\") pod \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\" (UID: \"17f7a4ec-12b8-473b-9cfd-2caed4a2e346\") " Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235394 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.235673 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.236167 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.236642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.236192 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.237461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.236666 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.237461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.236899 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.237813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.237776 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.245265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.245210 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:57.250065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.250038 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:57.250443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.250420 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "kube-api-access-5jsr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:57.251991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.251963 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy" pod "17f7a4ec-12b8-473b-9cfd-2caed4a2e346" (UID: "17f7a4ec-12b8-473b-9cfd-2caed4a2e346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.337645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337614 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337639 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337649 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337658 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337668 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jsr8\" (UniqueName: \"kubernetes.io/projected/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-kube-api-access-5jsr8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337677 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.337900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.337685 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f7a4ec-12b8-473b-9cfd-2caed4a2e346-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.980841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.980758 5058 generic.go:356] "Generic (PLEG): container finished" podID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerID="fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d" exitCode=0 Oct 30 05:21:57.980841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.980805 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" Oct 30 05:21:57.981352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.980830 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" event={"ID":"17f7a4ec-12b8-473b-9cfd-2caed4a2e346","Type":"ContainerDied","Data":"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d"} Oct 30 05:21:57.981352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.980889 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6" event={"ID":"17f7a4ec-12b8-473b-9cfd-2caed4a2e346","Type":"ContainerDied","Data":"197e487ffb1f52afea39267a5847ee78e1dd1aa0befb898e8cca5c61ac18983e"} Oct 30 05:21:57.981352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.980911 5058 scope.go:122] "RemoveContainer" containerID="fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d" Oct 30 05:21:57.992074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.992054 5058 scope.go:122] "RemoveContainer" containerID="fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d" Oct 30 05:21:57.992376 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:21:57.992357 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d\": container with ID starting with fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d not found: ID does not exist" containerID="fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d" Oct 30 05:21:57.992452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:57.992383 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d"} err="failed to get container status \"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d\": rpc error: code = NotFound desc = could not find container \"fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d\": container with ID starting with fdcfc9b84bedfa459a1039b7a4417e65c0678f1c0d0601ebaaa9e6fe1bfd7a1d not found: ID does not exist" Oct 30 05:21:58.051738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:58.051682 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:21:58.057776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:58.057747 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6"] Oct 30 05:21:59.050996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:59.050948 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:59.162426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:21:59.162391 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" path="/var/lib/kubelet/pods/17f7a4ec-12b8-473b-9cfd-2caed4a2e346/volumes" Oct 30 05:22:00.701778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.701746 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:22:00.702946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.702923 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" containerName="maas-api" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.702948 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" containerName="maas-api" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.702973 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d4047510-acbb-41cf-aa18-3817bd67037c" containerName="payload-processing" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.702981 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4047510-acbb-41cf-aa18-3817bd67037c" containerName="payload-processing" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703014 5058 cpu_manager.go:414] "containerMap: removing container" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703021 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" Oct 30 05:22:00.703041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703042 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerName="maas-api" Oct 30 05:22:00.703396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703050 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerName="maas-api" Oct 30 05:22:00.703396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703234 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef763ca4-ff7e-4b56-a1ee-93cbdd9dbaee" containerName="maas-api" Oct 30 05:22:00.703396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703266 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fe6ff0-d65c-4759-bf28-0f47cb9dddef" containerName="maas-api" Oct 30 05:22:00.703396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703278 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4047510-acbb-41cf-aa18-3817bd67037c" containerName="payload-processing" Oct 30 05:22:00.703396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.703293 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f7a4ec-12b8-473b-9cfd-2caed4a2e346" containerName="istio-proxy" Oct 30 05:22:00.709427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.709404 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.727690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.727667 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:22:00.733626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.733592 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"kube-root-ca.crt\"" Oct 30 05:22:00.733762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.733726 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"openshift-service-ca.crt\"" Oct 30 05:22:00.738788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.738760 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"sub-test-model-44bc69-kserve-self-signed-certs\"" Oct 30 05:22:00.738788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.738778 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"default-dockercfg-4vp7n\"" Oct 30 05:22:00.796301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796275 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.796440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796347 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6rv\" (UniqueName: \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.796440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796370 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.796440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796388 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.796587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796496 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.796587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.796531 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898279 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898364 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6rv\" (UniqueName: \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898387 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898403 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898473 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898511 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898742 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898861 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.898927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.898903 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.900861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.900835 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.900979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.900961 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:00.956328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:00.955026 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6rv\" (UniqueName: \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\") pod \"sub-test-model-44bc69-kserve-b8479bd9c-tx76r\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:01.036751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.036447 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:01.366696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.366611 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:22:01.371135 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:01.371099 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee7355e_e53b_4365_90e0_66b341d0f577.slice/crio-cd26d66d6dddd232d56e107f6dee494491f8e87026b9097c1f99b118a7c92c75 WatchSource:0}: Error finding container cd26d66d6dddd232d56e107f6dee494491f8e87026b9097c1f99b118a7c92c75: Status 404 returned error can't find the container with id cd26d66d6dddd232d56e107f6dee494491f8e87026b9097c1f99b118a7c92c75 Oct 30 05:22:01.750570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.750544 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:22:01.879133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879097 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879173 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879285 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879347 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879421 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879441 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:01.879609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879561 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879671 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879635 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879671 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879663 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:01.879784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879692 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879728 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\") pod \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\" (UID: \"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2\") " Oct 30 05:22:01.879995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.879971 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:01.880396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880309 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:01.880396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880364 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:01.880595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880574 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.880654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880601 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.880654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880616 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.880654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880629 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.880654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.880640 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.881651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.881630 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "kube-api-access-fh6ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:01.881742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.881671 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:01.881866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.881851 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:22:01.881940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.881926 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy" pod "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" (UID: "4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:01.982651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.982575 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6ld\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-kube-api-access-fh6ld\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.982651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.982605 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.982651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.982615 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.982651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:01.982623 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:02.068990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.068950 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" event={"ID":"fee7355e-e53b-4365-90e0-66b341d0f577","Type":"ContainerStarted","Data":"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799"} Oct 30 05:22:02.069196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.069002 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" event={"ID":"fee7355e-e53b-4365-90e0-66b341d0f577","Type":"ContainerStarted","Data":"cd26d66d6dddd232d56e107f6dee494491f8e87026b9097c1f99b118a7c92c75"} Oct 30 05:22:02.070692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.070669 5058 generic.go:356] "Generic (PLEG): container finished" podID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerID="d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef" exitCode=0 Oct 30 05:22:02.070779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.070706 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" Oct 30 05:22:02.070779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.070752 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerDied","Data":"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef"} Oct 30 05:22:02.070849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.070800 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f" event={"ID":"4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2","Type":"ContainerDied","Data":"5d7041948001c4751d8f8c6f764eaca5c6de33ad5ccf0baa7586e7711ab51af3"} Oct 30 05:22:02.070849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.070823 5058 scope.go:122] "RemoveContainer" containerID="d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef" Oct 30 05:22:02.082369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.082333 5058 scope.go:122] "RemoveContainer" containerID="c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b" Oct 30 05:22:02.092155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.092131 5058 scope.go:122] "RemoveContainer" containerID="d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef" Oct 30 05:22:02.092432 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:02.092413 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef\": container with ID starting with d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef not found: ID does not exist" containerID="d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef" Oct 30 05:22:02.092519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.092438 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef"} err="failed to get container status \"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef\": rpc error: code = NotFound desc = could not find container \"d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef\": container with ID starting with d85407190c9442c4b7776a0dbbdd54f95f1001c2331b03cff7b5c264e0ff12ef not found: ID does not exist" Oct 30 05:22:02.092519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.092456 5058 scope.go:122] "RemoveContainer" containerID="c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b" Oct 30 05:22:02.092683 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:02.092665 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b\": container with ID starting with c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b not found: ID does not exist" containerID="c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b" Oct 30 05:22:02.092740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.092685 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b"} err="failed to get container status \"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b\": rpc error: code = NotFound desc = could not find container \"c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b\": container with ID starting with c7f07341e3197d0fb588a2f4bb0212bcc93d76d473e312683f0d72c101088a4b not found: ID does not exist" Oct 30 05:22:02.112970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.112914 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" podStartSLOduration=2.11290086 podStartE2EDuration="2.11290086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:02.110282903 +0000 UTC m=+2295.079777920" watchObservedRunningTime="2026-10-30 05:22:02.11290086 +0000 UTC m=+2295.082395847" Oct 30 05:22:02.151486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.151451 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:22:02.155737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.155706 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f"] Oct 30 05:22:02.275509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.275418 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:22:02.276684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276657 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.276684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276685 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.276863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276705 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.276863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276712 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.276945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276883 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.276995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.276943 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" containerName="istio-proxy" Oct 30 05:22:02.284917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.284893 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:22:02.285061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.284990 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.304449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.301088 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-262c10-serving-cert\"" Oct 30 05:22:02.388071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.388041 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.388253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.388198 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5r9\" (UniqueName: \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.388318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.388264 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.388413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.388399 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.493238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.492428 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.493238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.492537 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.493238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.492701 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5r9\" (UniqueName: \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.493238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.492741 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.502814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.493659 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.504352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.503932 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.505433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.505398 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.515413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.515392 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5r9\" (UniqueName: \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\") pod \"maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.605448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.605372 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:02.908497 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:02.908464 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b0ee62_4ec2_47db_820e_531cc6eee35b.slice/crio-9479825776f755eca02e2f99328d5de8f5eee41a45f9455e99ed2b87d606e125 WatchSource:0}: Error finding container 9479825776f755eca02e2f99328d5de8f5eee41a45f9455e99ed2b87d606e125: Status 404 returned error can't find the container with id 9479825776f755eca02e2f99328d5de8f5eee41a45f9455e99ed2b87d606e125 Oct 30 05:22:02.909447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.909420 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:22:02.909934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:02.909918 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:22:03.039681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:03.039648 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:03.080485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:03.080449 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" event={"ID":"28b0ee62-4ec2-47db-820e-531cc6eee35b","Type":"ContainerStarted","Data":"9479825776f755eca02e2f99328d5de8f5eee41a45f9455e99ed2b87d606e125"} Oct 30 05:22:03.172271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:03.172174 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2" path="/var/lib/kubelet/pods/4e40f74e-f1d7-40e4-867d-5c5b6d7de9b2/volumes" Oct 30 05:22:04.089489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.089449 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" event={"ID":"28b0ee62-4ec2-47db-820e-531cc6eee35b","Type":"ContainerStarted","Data":"0ef8a6e8c6a8958d266b474ffa5336d14d34b428c558897338ec37a29d1953b3"} Oct 30 05:22:04.089910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.089556 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:04.131453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.131401 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" podStartSLOduration=1.739124362 podStartE2EDuration="2.131388067s" totalImagesPullingTime="392.263705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:04.130015167 +0000 UTC m=+2297.099510155" watchObservedRunningTime="2026-10-30 05:22:04.131388067 +0000 UTC m=+2297.100883055" Oct 30 05:22:04.781708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.781680 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:22:04.823664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.823618 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\") pod \"e73cd968-2ac9-4d7f-8d51-5de33781e059\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " Oct 30 05:22:04.823855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.823706 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\" (UniqueName: \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\") pod \"e73cd968-2ac9-4d7f-8d51-5de33781e059\" (UID: \"e73cd968-2ac9-4d7f-8d51-5de33781e059\") " Oct 30 05:22:04.824169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.824088 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume" pod "e73cd968-2ac9-4d7f-8d51-5de33781e059" (UID: "e73cd968-2ac9-4d7f-8d51-5de33781e059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:04.825013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.824935 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cd968-2ac9-4d7f-8d51-5de33781e059-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:04.830298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.826731 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl" pod "e73cd968-2ac9-4d7f-8d51-5de33781e059" (UID: "e73cd968-2ac9-4d7f-8d51-5de33781e059"). InnerVolumeSpecName "kube-api-access-cb4kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:04.925968 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:04.925928 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4kl\" (UniqueName: \"kubernetes.io/projected/e73cd968-2ac9-4d7f-8d51-5de33781e059-kube-api-access-cb4kl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:05.098993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.098900 5058 generic.go:356] "Generic (PLEG): container finished" podID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerID="f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6" exitCode=0 Oct 30 05:22:05.098993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.098946 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" Oct 30 05:22:05.098993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.098981 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" event={"ID":"e73cd968-2ac9-4d7f-8d51-5de33781e059","Type":"ContainerDied","Data":"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6"} Oct 30 05:22:05.099578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.099033 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn" event={"ID":"e73cd968-2ac9-4d7f-8d51-5de33781e059","Type":"ContainerDied","Data":"355bc9ea87f223bc6beb489ad0309efc30ca2a1bb13fd120308ffbf4a6b47b9b"} Oct 30 05:22:05.099578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.099055 5058 scope.go:122] "RemoveContainer" containerID="f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6" Oct 30 05:22:05.109995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.109975 5058 scope.go:122] "RemoveContainer" containerID="f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6" Oct 30 05:22:05.110294 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:05.110275 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6\": container with ID starting with f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6 not found: ID does not exist" containerID="f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6" Oct 30 05:22:05.110375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.110300 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6"} err="failed to get container status \"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6\": rpc error: code = NotFound desc = could not find container \"f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6\": container with ID starting with f1254e54b58d45ceec95deabf26a4c372627b15cb0e543de19d2a388f7d598f6 not found: ID does not exist" Oct 30 05:22:05.164894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.164862 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:22:05.169593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:05.169568 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn"] Oct 30 05:22:07.162728 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:07.162696 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" path="/var/lib/kubelet/pods/e73cd968-2ac9-4d7f-8d51-5de33781e059/volumes" Oct 30 05:22:10.105696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:10.105667 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:22:11.039207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:11.039176 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:11.051366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:11.051338 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:11.177162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:11.177132 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:22:17.008030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.007990 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:22:17.008979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.008956 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" Oct 30 05:22:17.008979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.008978 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" Oct 30 05:22:17.009144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.009133 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73cd968-2ac9-4d7f-8d51-5de33781e059" containerName="payload-pre-processing" Oct 30 05:22:17.012684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.012662 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.045562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.036539 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-0dba31-serving-cert\"" Oct 30 05:22:17.048345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.048316 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:22:17.146847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.146810 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.147033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.146861 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chln6\" (UniqueName: \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.147082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.147036 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.147276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.147204 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.261928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.261834 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.263915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.263646 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.265617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.265593 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.266398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.266378 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chln6\" (UniqueName: \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.266615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.266458 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.274943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.274482 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.277003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.276974 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.286087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.286058 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chln6\" (UniqueName: \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\") pod \"maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.330528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.330496 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:17.636701 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:17.636664 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7753b434_1229_4fe8_8848_560519c4028a.slice/crio-44b643f406562a1eaae927b334d1396e7bc7edacff277f8610a7cc8eca34b451 WatchSource:0}: Error finding container 44b643f406562a1eaae927b334d1396e7bc7edacff277f8610a7cc8eca34b451: Status 404 returned error can't find the container with id 44b643f406562a1eaae927b334d1396e7bc7edacff277f8610a7cc8eca34b451 Oct 30 05:22:17.638833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:17.638805 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:22:18.285409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:18.285331 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" event={"ID":"7753b434-1229-4fe8-8848-560519c4028a","Type":"ContainerStarted","Data":"6dd369c65e43317f7a57f9d1ab6fd23ed61d2ccfcf00c2a76491bb674a5b0cc0"} Oct 30 05:22:18.285409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:18.285386 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" event={"ID":"7753b434-1229-4fe8-8848-560519c4028a","Type":"ContainerStarted","Data":"44b643f406562a1eaae927b334d1396e7bc7edacff277f8610a7cc8eca34b451"} Oct 30 05:22:18.285912 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:18.285428 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:19.200150 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:19.200117 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:22:22.555497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:22.555441 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" podStartSLOduration=6.187650438 podStartE2EDuration="6.555426895s" totalImagesPullingTime="367.776457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:18.343637115 +0000 UTC m=+2311.313132104" watchObservedRunningTime="2026-10-30 05:22:22.555426895 +0000 UTC m=+2315.524921883" Oct 30 05:22:22.555932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:22.555621 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:22:22.588719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:22.588684 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:22:24.315149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:24.315113 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:22:27.556437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.556403 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" podUID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" containerName="payload-processing" containerID="cri-o://3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec" gracePeriod=25 Oct 30 05:22:27.589156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.589121 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" containerID="cri-o://78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb" gracePeriod=25 Oct 30 05:22:27.837921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.837634 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:22:27.945423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.945386 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\") pod \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " Oct 30 05:22:27.945613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.945508 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\" (UniqueName: \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\") pod \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\" (UID: \"72695511-cdb2-45dd-87c0-f5cdb6c70ae9\") " Oct 30 05:22:27.945756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.945732 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume" pod "72695511-cdb2-45dd-87c0-f5cdb6c70ae9" (UID: "72695511-cdb2-45dd-87c0-f5cdb6c70ae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:27.946063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.946042 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:27.947961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:27.947934 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p" pod "72695511-cdb2-45dd-87c0-f5cdb6c70ae9" (UID: "72695511-cdb2-45dd-87c0-f5cdb6c70ae9"). InnerVolumeSpecName "kube-api-access-cjt6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:28.047110 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.047084 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjt6p\" (UniqueName: \"kubernetes.io/projected/72695511-cdb2-45dd-87c0-f5cdb6c70ae9-kube-api-access-cjt6p\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:28.369663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.369633 5058 generic.go:356] "Generic (PLEG): container finished" podID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" containerID="3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec" exitCode=0 Oct 30 05:22:28.369840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.369682 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" event={"ID":"72695511-cdb2-45dd-87c0-f5cdb6c70ae9","Type":"ContainerDied","Data":"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec"} Oct 30 05:22:28.369840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.369720 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" event={"ID":"72695511-cdb2-45dd-87c0-f5cdb6c70ae9","Type":"ContainerDied","Data":"c91ecea437eb1120ed6c82af71e469b2d8feb231137b9ebd203dbeeb2a0dc7fb"} Oct 30 05:22:28.369840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.369742 5058 scope.go:122] "RemoveContainer" containerID="3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec" Oct 30 05:22:28.369840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.369684 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw" Oct 30 05:22:28.380681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.380666 5058 scope.go:122] "RemoveContainer" containerID="3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec" Oct 30 05:22:28.380936 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:28.380913 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec\": container with ID starting with 3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec not found: ID does not exist" containerID="3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec" Oct 30 05:22:28.381021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.380941 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec"} err="failed to get container status \"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec\": rpc error: code = NotFound desc = could not find container \"3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec\": container with ID starting with 3279f2b8906932cc95782eaefac7816f537a80b072fd91d787125fb38b8d2aec not found: ID does not exist" Oct 30 05:22:28.452336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.452311 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:22:28.455872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.455843 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw"] Oct 30 05:22:28.483591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:28.483561 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:29.162602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:29.162567 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" path="/var/lib/kubelet/pods/72695511-cdb2-45dd-87c0-f5cdb6c70ae9/volumes" Oct 30 05:22:35.861985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:35.861545 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:22:35.861985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:35.861791 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" podUID="54931758-280e-4ce8-80d0-ca8a689d0ea7" containerName="maas-api" containerID="cri-o://9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd" gracePeriod=30 Oct 30 05:22:36.256925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.256892 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:22:36.333950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.333915 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\") pod \"54931758-280e-4ce8-80d0-ca8a689d0ea7\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " Oct 30 05:22:36.334193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.334050 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\") pod \"54931758-280e-4ce8-80d0-ca8a689d0ea7\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " Oct 30 05:22:36.334193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.334088 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\" (UniqueName: \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\") pod \"54931758-280e-4ce8-80d0-ca8a689d0ea7\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " Oct 30 05:22:36.334193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.334183 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\") pod \"54931758-280e-4ce8-80d0-ca8a689d0ea7\" (UID: \"54931758-280e-4ce8-80d0-ca8a689d0ea7\") " Oct 30 05:22:36.335169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.335129 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca" pod "54931758-280e-4ce8-80d0-ca8a689d0ea7" (UID: "54931758-280e-4ce8-80d0-ca8a689d0ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:36.337490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.337467 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw" pod "54931758-280e-4ce8-80d0-ca8a689d0ea7" (UID: "54931758-280e-4ce8-80d0-ca8a689d0ea7"). InnerVolumeSpecName "kube-api-access-tlwpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:36.339888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.339865 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls" pod "54931758-280e-4ce8-80d0-ca8a689d0ea7" (UID: "54931758-280e-4ce8-80d0-ca8a689d0ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:36.340008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.339990 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs" pod "54931758-280e-4ce8-80d0-ca8a689d0ea7" (UID: "54931758-280e-4ce8-80d0-ca8a689d0ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:36.451303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.449961 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54931758-280e-4ce8-80d0-ca8a689d0ea7-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:36.451303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.450016 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:36.451303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.450033 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54931758-280e-4ce8-80d0-ca8a689d0ea7-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:36.451303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.450046 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlwpw\" (UniqueName: \"kubernetes.io/projected/54931758-280e-4ce8-80d0-ca8a689d0ea7-kube-api-access-tlwpw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:36.484806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.484768 5058 generic.go:356] "Generic (PLEG): container finished" podID="54931758-280e-4ce8-80d0-ca8a689d0ea7" containerID="9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd" exitCode=0 Oct 30 05:22:36.486076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.486041 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" event={"ID":"54931758-280e-4ce8-80d0-ca8a689d0ea7","Type":"ContainerDied","Data":"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd"} Oct 30 05:22:36.486421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.486393 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" event={"ID":"54931758-280e-4ce8-80d0-ca8a689d0ea7","Type":"ContainerDied","Data":"fb2573b41ed1a9d192fc51cf3f88a4fcd2689ca23d1770b527d9feab75c6577e"} Oct 30 05:22:36.486553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.486439 5058 scope.go:122] "RemoveContainer" containerID="9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd" Oct 30 05:22:36.489518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.486925 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm" Oct 30 05:22:36.521562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.521533 5058 scope.go:122] "RemoveContainer" containerID="9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd" Oct 30 05:22:36.523924 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:36.523636 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd\": container with ID starting with 9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd not found: ID does not exist" containerID="9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd" Oct 30 05:22:36.523924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.523679 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd"} err="failed to get container status \"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd\": rpc error: code = NotFound desc = could not find container \"9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd\": container with ID starting with 9f201b6aec353ce22ba5d5c8552b7ccab877e44dc6293063b7d95cb631deabfd not found: ID does not exist" Oct 30 05:22:36.552120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.552090 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:22:36.559866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:36.559841 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8f18-7ccf469548-sjntm"] Oct 30 05:22:37.161960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:37.161929 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54931758-280e-4ce8-80d0-ca8a689d0ea7" path="/var/lib/kubelet/pods/54931758-280e-4ce8-80d0-ca8a689d0ea7/volumes" Oct 30 05:22:38.483667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:38.483621 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:41.840941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.840910 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:22:41.841709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.841690 5058 cpu_manager.go:414] "containerMap: removing container" podUID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" containerName="payload-processing" Oct 30 05:22:41.841709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.841708 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" containerName="payload-processing" Oct 30 05:22:41.841936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.841757 5058 cpu_manager.go:414] "containerMap: removing container" podUID="54931758-280e-4ce8-80d0-ca8a689d0ea7" containerName="maas-api" Oct 30 05:22:41.841936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.841764 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54931758-280e-4ce8-80d0-ca8a689d0ea7" containerName="maas-api" Oct 30 05:22:41.842474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.842455 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="72695511-cdb2-45dd-87c0-f5cdb6c70ae9" containerName="payload-processing" Oct 30 05:22:41.842595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.842510 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="54931758-280e-4ce8-80d0-ca8a689d0ea7" containerName="maas-api" Oct 30 05:22:41.848473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.848448 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.869655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.868460 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:22:41.874986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.874959 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"sub-test-model-93bad9-kserve-self-signed-certs\"" Oct 30 05:22:41.875142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.875038 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"openshift-service-ca.crt\"" Oct 30 05:22:41.879841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.879810 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"kube-root-ca.crt\"" Oct 30 05:22:41.880123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.880103 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"default-dockercfg-h552l\"" Oct 30 05:22:41.920821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.920792 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.920980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.920920 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4ld\" (UniqueName: \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.920980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.920967 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.921126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.921109 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.921207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.921193 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:41.921265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:41.921235 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022421 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022483 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022503 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022637 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4ld\" (UniqueName: \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.022782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022662 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.023014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.022977 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.023108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.023033 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.023146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.023133 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.024749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.024727 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.024946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.024927 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.044125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.044104 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4ld\" (UniqueName: \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\") pod \"sub-test-model-93bad9-kserve-55685849bf-cfpnz\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.169541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.169496 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:42.505030 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:42.505001 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec4a9e9f_9ebc_47b4_88ff_e581f2175fde.slice/crio-816f826f68e6408daddb09f5dc8ad9a0f06dd6769941c849de058fc98af1b5fa WatchSource:0}: Error finding container 816f826f68e6408daddb09f5dc8ad9a0f06dd6769941c849de058fc98af1b5fa: Status 404 returned error can't find the container with id 816f826f68e6408daddb09f5dc8ad9a0f06dd6769941c849de058fc98af1b5fa Oct 30 05:22:42.505718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.505694 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:22:42.546650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:42.546598 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" event={"ID":"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde","Type":"ContainerStarted","Data":"816f826f68e6408daddb09f5dc8ad9a0f06dd6769941c849de058fc98af1b5fa"} Oct 30 05:22:43.555009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:43.554972 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" event={"ID":"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde","Type":"ContainerStarted","Data":"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4"} Oct 30 05:22:43.602187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:43.602123 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" podStartSLOduration=2.6021061100000002 podStartE2EDuration="2.60210611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:43.599751762 +0000 UTC m=+2336.569246755" watchObservedRunningTime="2026-10-30 05:22:43.60210611 +0000 UTC m=+2336.571601099" Oct 30 05:22:43.653559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:43.653524 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:22:43.653765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:43.653728 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" containerID="cri-o://2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2" gracePeriod=30 Oct 30 05:22:44.171719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:44.171687 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:44.248188 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:44.248148 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:22:46.145078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:46.145026 5058 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:22:46.145600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:46.145110 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:22:47.666731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:47.666700 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:22:47.667151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:47.666953 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" podUID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" containerName="maas-api" containerID="cri-o://d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813" gracePeriod=30 Oct 30 05:22:47.998477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:47.998128 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:22:48.047241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.047195 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:22:48.141636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.141585 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\") pod \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " Oct 30 05:22:48.141840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.141663 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\") pod \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " Oct 30 05:22:48.141840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.141794 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\" (UniqueName: \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\") pod \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " Oct 30 05:22:48.141840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.141827 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\") pod \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " Oct 30 05:22:48.142017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.141967 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") pod \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\" (UID: \"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac\") " Oct 30 05:22:48.142075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.142032 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\" (UniqueName: \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\") pod \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\" (UID: \"b0798140-c1d7-4bb9-aaa0-f154c04e5f30\") " Oct 30 05:22:48.142440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.142032 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca" pod "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" (UID: "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:48.142655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.142628 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.145341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.142828 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume" pod "b0798140-c1d7-4bb9-aaa0-f154c04e5f30" (UID: "b0798140-c1d7-4bb9-aaa0-f154c04e5f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:48.145341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.144498 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs" pod "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" (UID: "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:48.145341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.144934 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls" pod "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" (UID: "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:48.145341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.145105 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb" pod "b0798140-c1d7-4bb9-aaa0-f154c04e5f30" (UID: "b0798140-c1d7-4bb9-aaa0-f154c04e5f30"). InnerVolumeSpecName "kube-api-access-zh7xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:48.145751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.145729 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw" pod "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" (UID: "f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac"). InnerVolumeSpecName "kube-api-access-d5gpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:48.250320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.244546 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.250320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.244584 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7xb\" (UniqueName: \"kubernetes.io/projected/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-kube-api-access-zh7xb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.250320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.244599 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.250320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.244614 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5gpw\" (UniqueName: \"kubernetes.io/projected/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac-kube-api-access-d5gpw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.250320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.244630 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0798140-c1d7-4bb9-aaa0-f154c04e5f30-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:48.595272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.595179 5058 generic.go:356] "Generic (PLEG): container finished" podID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerID="78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb" exitCode=0 Oct 30 05:22:48.595272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.595237 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" Oct 30 05:22:48.595272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.595254 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" event={"ID":"b0798140-c1d7-4bb9-aaa0-f154c04e5f30","Type":"ContainerDied","Data":"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb"} Oct 30 05:22:48.595559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.595321 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg" event={"ID":"b0798140-c1d7-4bb9-aaa0-f154c04e5f30","Type":"ContainerDied","Data":"a866b7a33f4b111f09b2046984a546234a90294a240374679204ac41551d6018"} Oct 30 05:22:48.595559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.595351 5058 scope.go:122] "RemoveContainer" containerID="78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb" Oct 30 05:22:48.596846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.596803 5058 generic.go:356] "Generic (PLEG): container finished" podID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" containerID="d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813" exitCode=0 Oct 30 05:22:48.596846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.596837 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" event={"ID":"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac","Type":"ContainerDied","Data":"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813"} Oct 30 05:22:48.597058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.596867 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" Oct 30 05:22:48.597058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.596884 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr" event={"ID":"f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac","Type":"ContainerDied","Data":"2565b97667aebed9e7948ab3025a18c0bdb3f9982d64a168594d9bd7d7f6b2dd"} Oct 30 05:22:48.620794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.620770 5058 scope.go:122] "RemoveContainer" containerID="78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb" Oct 30 05:22:48.621086 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:48.621067 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb\": container with ID starting with 78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb not found: ID does not exist" containerID="78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb" Oct 30 05:22:48.621192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.621091 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb"} err="failed to get container status \"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb\": rpc error: code = NotFound desc = could not find container \"78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb\": container with ID starting with 78009df23cbc7b025f89f7c5815934bd3b23430d5495dae1819bc608268353fb not found: ID does not exist" Oct 30 05:22:48.621192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.621110 5058 scope.go:122] "RemoveContainer" containerID="d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813" Oct 30 05:22:48.632486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.632468 5058 scope.go:122] "RemoveContainer" containerID="d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813" Oct 30 05:22:48.632752 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:48.632731 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813\": container with ID starting with d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813 not found: ID does not exist" containerID="d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813" Oct 30 05:22:48.632808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.632759 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813"} err="failed to get container status \"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813\": rpc error: code = NotFound desc = could not find container \"d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813\": container with ID starting with d3b0b2764101b8173304d4f66c9c5738dfe222f6047071bedea0b1523ea76813 not found: ID does not exist" Oct 30 05:22:48.671134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.671106 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:22:48.676359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.676335 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-356cae-7676d59b5b-nx5wr"] Oct 30 05:22:48.711310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.711285 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:22:48.714716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.714693 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg"] Oct 30 05:22:48.952634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:48.952611 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:22:49.058155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058118 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058197 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058237 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058296 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058339 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058455 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058469 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.058603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058489 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058532 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.058814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058649 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.058814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.058675 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\") pod \"3b8463a0-07eb-4ece-aaee-6867150eafca\" (UID: \"3b8463a0-07eb-4ece-aaee-6867150eafca\") " Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059021 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059398 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059421 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059438 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059440 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.059483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.059445 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:49.061168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.061136 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:22:49.061607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.061581 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:49.061712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.061663 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.062395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.062370 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph" pod "3b8463a0-07eb-4ece-aaee-6867150eafca" (UID: "3b8463a0-07eb-4ece-aaee-6867150eafca"). InnerVolumeSpecName "kube-api-access-l8nph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:49.160585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160551 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.160585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160579 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8463a0-07eb-4ece-aaee-6867150eafca-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.160585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160589 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.160824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160598 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8463a0-07eb-4ece-aaee-6867150eafca-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.160824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160607 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8nph\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-kube-api-access-l8nph\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.160824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.160620 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8463a0-07eb-4ece-aaee-6867150eafca-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.163151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.163124 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" path="/var/lib/kubelet/pods/b0798140-c1d7-4bb9-aaa0-f154c04e5f30/volumes" Oct 30 05:22:49.163636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.163613 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" path="/var/lib/kubelet/pods/f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac/volumes" Oct 30 05:22:49.194270 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:49.194200 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.236165 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237294 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237316 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237341 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237349 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237386 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" containerName="maas-api" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237394 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" containerName="maas-api" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237612 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerName="istio-proxy" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237631 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0798140-c1d7-4bb9-aaa0-f154c04e5f30" containerName="payload-pre-processing" Oct 30 05:22:49.238272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.237646 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b7ed6d-0bbb-421f-a02c-0c4fe6ced4ac" containerName="maas-api" Oct 30 05:22:49.247047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.247003 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.259909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.259880 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:22:49.266332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.266308 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6b3855-openshift-default-dockercfg-vbtxb\"" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378211 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378320 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378381 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378489 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378533 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378558 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378592 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378634 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbbm\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.455330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.378683 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490171 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490328 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490386 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490415 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490449 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490499 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbbm\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490539 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490718 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.491605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.490812 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.494331 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.494643 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.494905 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.495131 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.495399 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.499252 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.508291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.505932 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.523105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.523042 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.548304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.538407 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbbm\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.577910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.577885 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:49.611822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.611780 5058 generic.go:356] "Generic (PLEG): container finished" podID="3b8463a0-07eb-4ece-aaee-6867150eafca" containerID="2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2" exitCode=0 Oct 30 05:22:49.611943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.611834 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" Oct 30 05:22:49.611943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.611860 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" event={"ID":"3b8463a0-07eb-4ece-aaee-6867150eafca","Type":"ContainerDied","Data":"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2"} Oct 30 05:22:49.611943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.611937 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw" event={"ID":"3b8463a0-07eb-4ece-aaee-6867150eafca","Type":"ContainerDied","Data":"5a4a8fdb5aae0b66da13409d9b15672b8c7d409fb6688b4d9c32ac1704d33da2"} Oct 30 05:22:49.612104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.611973 5058 scope.go:122] "RemoveContainer" containerID="2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2" Oct 30 05:22:49.637271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.637119 5058 scope.go:122] "RemoveContainer" containerID="2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2" Oct 30 05:22:49.637791 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:49.637763 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2\": container with ID starting with 2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2 not found: ID does not exist" containerID="2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2" Oct 30 05:22:49.637893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.637801 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2"} err="failed to get container status \"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2\": rpc error: code = NotFound desc = could not find container \"2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2\": container with ID starting with 2073ad02664cb725aec752d80b44a93cadbbdbe7a15905d667162f358a98dcf2 not found: ID does not exist" Oct 30 05:22:49.676412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.676371 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:22:49.688601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.688322 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw"] Oct 30 05:22:49.944406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.942521 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:22:49.955484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.955445 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:49.955581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.955521 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:49.955581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:49.955564 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:50.623124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:50.623084 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerStarted","Data":"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f"} Oct 30 05:22:50.623351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:50.623151 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerStarted","Data":"175625dafbfac70d9c2d6ebb884fda93c66c024fb85cd65cc2eb136a51c5eff5"} Oct 30 05:22:50.669263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:50.669201 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podStartSLOduration=1.66918342 podStartE2EDuration="1.66918342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:50.667040451 +0000 UTC m=+2343.636535452" watchObservedRunningTime="2026-10-30 05:22:50.66918342 +0000 UTC m=+2343.638678409" Oct 30 05:22:51.164093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:51.164060 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8463a0-07eb-4ece-aaee-6867150eafca" path="/var/lib/kubelet/pods/3b8463a0-07eb-4ece-aaee-6867150eafca/volumes" Oct 30 05:22:51.578776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:51.578696 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:51.584769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:51.584742 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:51.631021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:51.630989 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:51.631912 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:51.631889 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:22:52.171873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:52.171828 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:52.187449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:52.187422 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:52.648837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:52.648807 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:22:54.124793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.124764 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:22:54.130501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.130478 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.155659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.155626 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:22:54.162122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.162088 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-674082\"" Oct 30 05:22:54.209350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.200989 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.209350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.201204 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxsp\" (UniqueName: \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.233064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.227625 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-674082-dockercfg-nnb6h\"" Oct 30 05:22:54.316252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.313838 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:22:54.327249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.319632 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:22:54.327249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.319749 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.372591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.369183 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxsp\" (UniqueName: \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.372591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.369261 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gtv\" (UniqueName: \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.372591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.369405 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.372591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.369474 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.372591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.371144 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.479042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.474211 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gtv\" (UniqueName: \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.479042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.474529 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.479042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.475743 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.493623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.493571 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxsp\" (UniqueName: \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\") pod \"payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.498954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.498922 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gtv\" (UniqueName: \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\") pod \"payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:54.756179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.756086 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:54.794584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:54.794552 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:55.070812 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:55.070782 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef450b5_d521_44f0_b2b0_4808c3f503d8.slice/crio-41583137ff38fc3cd68020a934205694ef876ec915c3a0515a42ca77a3f3c0e6 WatchSource:0}: Error finding container 41583137ff38fc3cd68020a934205694ef876ec915c3a0515a42ca77a3f3c0e6: Status 404 returned error can't find the container with id 41583137ff38fc3cd68020a934205694ef876ec915c3a0515a42ca77a3f3c0e6 Oct 30 05:22:55.072620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.072592 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:22:55.132377 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:55.132343 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7e3e12_78ac_427c_9790_004b30634d4d.slice/crio-2b8592ee55835e7662f0d6a42989468db58f3351f1dedb693b69461d1a1fc606 WatchSource:0}: Error finding container 2b8592ee55835e7662f0d6a42989468db58f3351f1dedb693b69461d1a1fc606: Status 404 returned error can't find the container with id 2b8592ee55835e7662f0d6a42989468db58f3351f1dedb693b69461d1a1fc606 Oct 30 05:22:55.132850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.132836 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:22:55.665122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.665084 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" event={"ID":"5c7e3e12-78ac-427c-9790-004b30634d4d","Type":"ContainerStarted","Data":"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c"} Oct 30 05:22:55.665319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.665147 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" event={"ID":"5c7e3e12-78ac-427c-9790-004b30634d4d","Type":"ContainerStarted","Data":"2b8592ee55835e7662f0d6a42989468db58f3351f1dedb693b69461d1a1fc606"} Oct 30 05:22:55.665319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.665174 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:22:55.666756 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.666728 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" event={"ID":"5ef450b5-d521-44f0-b2b0-4808c3f503d8","Type":"ContainerStarted","Data":"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef"} Oct 30 05:22:55.666895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.666775 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" event={"ID":"5ef450b5-d521-44f0-b2b0-4808c3f503d8","Type":"ContainerStarted","Data":"41583137ff38fc3cd68020a934205694ef876ec915c3a0515a42ca77a3f3c0e6"} Oct 30 05:22:55.666895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.666804 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:22:55.719078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.719011 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" podStartSLOduration=1.718991232 podStartE2EDuration="1.718991232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:55.713366844 +0000 UTC m=+2348.682861833" watchObservedRunningTime="2026-10-30 05:22:55.718991232 +0000 UTC m=+2348.688486222" Oct 30 05:22:55.756581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.756532 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" podStartSLOduration=1.7565181079999999 podStartE2EDuration="1.756518108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:55.756096853 +0000 UTC m=+2348.725591844" watchObservedRunningTime="2026-10-30 05:22:55.756518108 +0000 UTC m=+2348.726013095" Oct 30 05:22:55.804873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.804842 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:22:55.809778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.809756 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:55.825709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.825681 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3855\"" Oct 30 05:22:55.825847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.825759 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3855-dockercfg-h5gxz\"" Oct 30 05:22:55.827699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.827672 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:22:55.894193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.894154 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz9v\" (UniqueName: \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:55.894418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:55.894388 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.027288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.024453 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz9v\" (UniqueName: \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.027288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.024652 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.106271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.106062 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz9v\" (UniqueName: \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.108250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.107529 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.128244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.127520 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.448084 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:22:56.448053 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c4a1d0_7f84_4834_97f6_d890280f7257.slice/crio-f537d57a651fc2407e18a783c0492f6708598cb980777050dd227b978635d6f8 WatchSource:0}: Error finding container f537d57a651fc2407e18a783c0492f6708598cb980777050dd227b978635d6f8: Status 404 returned error can't find the container with id f537d57a651fc2407e18a783c0492f6708598cb980777050dd227b978635d6f8 Oct 30 05:22:56.449139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.449114 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:22:56.675015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.674965 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" event={"ID":"c5c4a1d0-7f84-4834-97f6-d890280f7257","Type":"ContainerStarted","Data":"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6"} Oct 30 05:22:56.675180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.675030 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" event={"ID":"c5c4a1d0-7f84-4834-97f6-d890280f7257","Type":"ContainerStarted","Data":"f537d57a651fc2407e18a783c0492f6708598cb980777050dd227b978635d6f8"} Oct 30 05:22:56.675180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.675060 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:22:56.718206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:56.718117 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" podStartSLOduration=1.718101316 podStartE2EDuration="1.718101316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:56.717333991 +0000 UTC m=+2349.686829014" watchObservedRunningTime="2026-10-30 05:22:56.718101316 +0000 UTC m=+2349.687596302" Oct 30 05:22:59.271292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.271249 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerName="main" containerID="cri-o://4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9" gracePeriod=45 Oct 30 05:22:59.412785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.412744 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": dial tcp 10.132.0.181:8000: connect: connection refused" Oct 30 05:22:59.552554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.552528 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:22:59.602402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602372 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\" (UniqueName: \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602498 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602676 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602800 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602850 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602899 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\") pod \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\" (UID: \"d1ea2a04-6082-4c07-99e5-8b0b70e24207\") " Oct 30 05:22:59.602966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.602943 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:59.603154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.603134 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:59.603577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.603550 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.603577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.603578 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.603812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.603791 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:59.605282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.605090 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "kube-api-access-n2j2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:59.605459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.605435 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:59.605769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.605744 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm" pod "d1ea2a04-6082-4c07-99e5-8b0b70e24207" (UID: "d1ea2a04-6082-4c07-99e5-8b0b70e24207"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:59.698990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.698956 5058 generic.go:356] "Generic (PLEG): container finished" podID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerID="4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9" exitCode=0 Oct 30 05:22:59.699171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.699001 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" Oct 30 05:22:59.699171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.699041 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" event={"ID":"d1ea2a04-6082-4c07-99e5-8b0b70e24207","Type":"ContainerDied","Data":"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9"} Oct 30 05:22:59.699171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.699083 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p" event={"ID":"d1ea2a04-6082-4c07-99e5-8b0b70e24207","Type":"ContainerDied","Data":"f56450b6d1631b927ee494cff904246a3f52f258825d00ae21d80f5e13dd6c8d"} Oct 30 05:22:59.699171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.699103 5058 scope.go:122] "RemoveContainer" containerID="4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9" Oct 30 05:22:59.705547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.705522 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.705547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.705547 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ea2a04-6082-4c07-99e5-8b0b70e24207-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.705706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.705557 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2j2t\" (UniqueName: \"kubernetes.io/projected/d1ea2a04-6082-4c07-99e5-8b0b70e24207-kube-api-access-n2j2t\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.705706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.705570 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ea2a04-6082-4c07-99e5-8b0b70e24207-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:22:59.710954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.710937 5058 scope.go:122] "RemoveContainer" containerID="4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9" Oct 30 05:22:59.711260 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:22:59.711213 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9\": container with ID starting with 4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9 not found: ID does not exist" containerID="4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9" Oct 30 05:22:59.711345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.711265 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9"} err="failed to get container status \"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9\": rpc error: code = NotFound desc = could not find container \"4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9\": container with ID starting with 4a398537523daf5b2374263e9a51f3b432cec1decbcb54bc8eb3ff5d996476e9 not found: ID does not exist" Oct 30 05:22:59.754480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.754456 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:22:59.757899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:22:59.757879 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2f6fed/auth-test-model-2f6fed-kserve-7ddf946457-8t56p"] Oct 30 05:23:01.164000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:01.163961 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" path="/var/lib/kubelet/pods/d1ea2a04-6082-4c07-99e5-8b0b70e24207/volumes" Oct 30 05:23:06.678207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:06.678177 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:23:06.678708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:06.678280 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:23:07.686546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:07.686514 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:23:14.908235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.908185 5058 generic.go:356] "Generic (PLEG): container finished" podID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerID="e5e341063ceb9880064af3f2334fe534768fdf90d16f13619f3886ac02cbd8f9" exitCode=0 Oct 30 05:23:14.908777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.908261 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerDied","Data":"e5e341063ceb9880064af3f2334fe534768fdf90d16f13619f3886ac02cbd8f9"} Oct 30 05:23:14.908850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.908820 5058 scope.go:122] "RemoveContainer" containerID="e5e341063ceb9880064af3f2334fe534768fdf90d16f13619f3886ac02cbd8f9" Oct 30 05:23:14.910777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.910750 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:14.910878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.910818 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:14.910878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:14.910852 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:15.918406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:15.918369 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerStarted","Data":"82d4c0e75fca3cab04623c81a56a2e08d068cac43c3900efdcd8957a6bbfd2af"} Oct 30 05:23:16.447242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:16.424253 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:23:16.447431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:16.447387 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:16.447530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:16.447462 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:17.413008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:17.412974 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:17.413426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:17.413023 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:18.412554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:18.412514 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:18.412795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:18.412583 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:19.195210 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:23:19.195174 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:23:19.412928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:19.412893 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:19.413135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:19.412945 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:20.413107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:20.413074 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:20.413524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:20.413124 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:21.412984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:21.412943 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:23:21.413169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:21.413012 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:23:22.270625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:22.270595 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:23:22.470923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:22.470893 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:23:22.989511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:22.989479 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:23:23.577368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.577336 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:23:23.578123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.578106 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerName="main" Oct 30 05:23:23.578173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.578124 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerName="main" Oct 30 05:23:23.578248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.578237 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ea2a04-6082-4c07-99e5-8b0b70e24207" containerName="main" Oct 30 05:23:23.583007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.582982 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.603718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.603693 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:23:23.612129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.612099 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-93bad9-openshift-default-dockercfg-hnpf6\"" Oct 30 05:23:23.622227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622182 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnlc\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622334 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622453 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622385 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622451 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622527 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622552 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622579 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622843 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.622971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.622924 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.723858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.723826 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.723868 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.723922 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.723973 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.723997 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724104 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724265 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724300 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724330 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724388 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724430 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnlc\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724529 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724751 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.724861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.724830 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.726811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.726791 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.726811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.726801 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.746598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.746575 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.746816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.746797 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnlc\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:23.897293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:23.897257 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:24.228648 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:23:24.228618 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce31c438_647b_4639_b235_f2073e8e0467.slice/crio-178aef0bcb63856719f64cb339da5a3dd6f62d30090e04409a3c8dd453a0b9c4 WatchSource:0}: Error finding container 178aef0bcb63856719f64cb339da5a3dd6f62d30090e04409a3c8dd453a0b9c4: Status 404 returned error can't find the container with id 178aef0bcb63856719f64cb339da5a3dd6f62d30090e04409a3c8dd453a0b9c4 Oct 30 05:23:24.229955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:24.229926 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:23:24.231681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:24.231646 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:24.231769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:24.231707 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:24.231769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:24.231736 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:25.005504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.005459 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" event={"ID":"ce31c438-647b-4639-b235-f2073e8e0467","Type":"ContainerStarted","Data":"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff"} Oct 30 05:23:25.005906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.005532 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" event={"ID":"ce31c438-647b-4639-b235-f2073e8e0467","Type":"ContainerStarted","Data":"178aef0bcb63856719f64cb339da5a3dd6f62d30090e04409a3c8dd453a0b9c4"} Oct 30 05:23:25.067964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.067897 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podStartSLOduration=2.067880456 podStartE2EDuration="2.067880456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:25.065182706 +0000 UTC m=+2378.034677694" watchObservedRunningTime="2026-10-30 05:23:25.067880456 +0000 UTC m=+2378.037375445" Oct 30 05:23:25.899359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.899321 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:25.903366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.903331 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 05:23:25.903516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:25.903384 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 05:23:26.898205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:26.898171 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 05:23:26.898602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:26.898294 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 05:23:27.898600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:27.898522 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 05:23:27.898600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:27.898572 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 05:23:28.898201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:28.898161 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 05:23:28.898379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:28.898239 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 05:23:29.897782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:29.897748 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 05:23:29.898180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:29.897797 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 05:23:30.902511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:30.902474 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:30.902975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:30.902954 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:30.903545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:30.903523 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:23:32.953231 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:32.953178 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:23:32.959339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:32.959317 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:32.978076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:32.978046 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:23:32.987687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:32.987410 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-674082-serving-cert\"" Oct 30 05:23:33.017929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.017895 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.018123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.017940 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xc7n\" (UniqueName: \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.018123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.017990 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.018123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.018068 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.119525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.119487 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.119751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.119607 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.119751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.119644 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xc7n\" (UniqueName: \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.119751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.119718 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.120419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.120393 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.122281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.122252 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.122479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.122455 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.140835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.140804 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xc7n\" (UniqueName: \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\") pod \"maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.275572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.275492 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:33.584732 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:23:33.584702 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabfd824_40fe_45f2_86cf_acb23b2f9d07.slice/crio-f95b8fc77170f9b6dacffba4f6ce632772146cb2aaa6683bf9425e143c8aebc7 WatchSource:0}: Error finding container f95b8fc77170f9b6dacffba4f6ce632772146cb2aaa6683bf9425e143c8aebc7: Status 404 returned error can't find the container with id f95b8fc77170f9b6dacffba4f6ce632772146cb2aaa6683bf9425e143c8aebc7 Oct 30 05:23:33.586906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:33.586879 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:23:34.106168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.106119 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" event={"ID":"dabfd824-40fe-45f2-86cf-acb23b2f9d07","Type":"ContainerStarted","Data":"8ca3c0333908de3dfc88ebd4087588122aa46c3d48bd45740ef314acc80e0dc8"} Oct 30 05:23:34.106694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.106191 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" event={"ID":"dabfd824-40fe-45f2-86cf-acb23b2f9d07","Type":"ContainerStarted","Data":"f95b8fc77170f9b6dacffba4f6ce632772146cb2aaa6683bf9425e143c8aebc7"} Oct 30 05:23:34.106694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.106366 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:34.149127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.149067 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" podStartSLOduration=1.7522183340000002 podStartE2EDuration="2.149054655s" totalImagesPullingTime="396.836321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:34.147084475 +0000 UTC m=+2387.116579473" watchObservedRunningTime="2026-10-30 05:23:34.149054655 +0000 UTC m=+2387.118549642" Oct 30 05:23:34.797421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.797385 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:23:34.802207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.802182 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.827554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.821074 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:23:34.828786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.828765 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6b3855-serving-cert\"" Oct 30 05:23:34.848607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.848574 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.848766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.848653 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.848839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.848775 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.848839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.848811 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnbq\" (UniqueName: \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.950757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.950724 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.950952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.950869 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.950952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.950910 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.950952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.950929 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnbq\" (UniqueName: \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.951617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.951580 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.953293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.953268 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.953451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.953430 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:34.971838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:34.971813 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnbq\" (UniqueName: \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\") pod \"maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:35.118740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:35.118638 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:35.422078 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:23:35.422047 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26397e20_d69e_4c5e_bc48_861c36272a84.slice/crio-e47bfee22156dc604f7e37ccdac805b84ab64d543b13c246e0e50595a89f6217 WatchSource:0}: Error finding container e47bfee22156dc604f7e37ccdac805b84ab64d543b13c246e0e50595a89f6217: Status 404 returned error can't find the container with id e47bfee22156dc604f7e37ccdac805b84ab64d543b13c246e0e50595a89f6217 Oct 30 05:23:35.425584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:35.425554 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:23:36.125776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:36.125686 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" event={"ID":"26397e20-d69e-4c5e-bc48-861c36272a84","Type":"ContainerStarted","Data":"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2"} Oct 30 05:23:36.125776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:36.125768 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" event={"ID":"26397e20-d69e-4c5e-bc48-861c36272a84","Type":"ContainerStarted","Data":"e47bfee22156dc604f7e37ccdac805b84ab64d543b13c246e0e50595a89f6217"} Oct 30 05:23:36.126286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:36.125936 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:36.202241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:36.196238 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" podStartSLOduration=1.888604796 podStartE2EDuration="2.196203605s" totalImagesPullingTime="307.598809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:36.195852203 +0000 UTC m=+2389.165347192" watchObservedRunningTime="2026-10-30 05:23:36.196203605 +0000 UTC m=+2389.165698592" Oct 30 05:23:40.122127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:40.122098 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:23:42.181398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:42.181366 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:23:45.295434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.295400 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:23:45.296001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.295632 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" podUID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" containerName="maas-api" containerID="cri-o://0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4" gracePeriod=30 Oct 30 05:23:45.819433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.819399 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:23:45.870985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.870950 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\") pod \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " Oct 30 05:23:45.872093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.872063 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\") pod \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " Oct 30 05:23:45.872348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.872324 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\") pod \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " Oct 30 05:23:45.872589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.872576 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\" (UniqueName: \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\") pod \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\" (UID: \"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2\") " Oct 30 05:23:45.877263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.872077 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca" pod "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" (UID: "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:45.886440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.886411 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264" pod "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" (UID: "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2"). InnerVolumeSpecName "kube-api-access-r8264". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:45.886708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.886688 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls" pod "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" (UID: "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:45.888276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.887942 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs" pod "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" (UID: "6842bad2-fc0e-49fb-9e5f-649ff4ef60a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:45.975167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.975080 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8264\" (UniqueName: \"kubernetes.io/projected/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-kube-api-access-r8264\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.975167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.975115 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.975517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.975173 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.975517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:45.975189 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:46.292118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.292024 5058 generic.go:356] "Generic (PLEG): container finished" podID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" containerID="0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4" exitCode=0 Oct 30 05:23:46.292118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.292070 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" Oct 30 05:23:46.292409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.292112 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" event={"ID":"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2","Type":"ContainerDied","Data":"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4"} Oct 30 05:23:46.292409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.292168 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg" event={"ID":"6842bad2-fc0e-49fb-9e5f-649ff4ef60a2","Type":"ContainerDied","Data":"3b76b70f82f99e9342f74b1f10b77a78269a740fd41793c4b89270b5d83640d5"} Oct 30 05:23:46.292409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.292200 5058 scope.go:122] "RemoveContainer" containerID="0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4" Oct 30 05:23:46.306307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.306116 5058 scope.go:122] "RemoveContainer" containerID="0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4" Oct 30 05:23:46.306601 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:23:46.306469 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4\": container with ID starting with 0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4 not found: ID does not exist" containerID="0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4" Oct 30 05:23:46.306601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.306504 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4"} err="failed to get container status \"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4\": rpc error: code = NotFound desc = could not find container \"0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4\": container with ID starting with 0de1661d6b671b3a18d8ca9994c820fed67f94330b856cd3420429a1f000dba4 not found: ID does not exist" Oct 30 05:23:46.348113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.348076 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:23:46.361086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:46.361056 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e16297-7dcfddf69b-tstsg"] Oct 30 05:23:47.164310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:47.164277 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" path="/var/lib/kubelet/pods/6842bad2-fc0e-49fb-9e5f-649ff4ef60a2/volumes" Oct 30 05:23:47.655874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:47.655843 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:23:47.669460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:47.669437 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:23:49.195270 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:23:49.195206 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:23:53.357035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:53.357002 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:23:53.357567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:53.357270 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" containerID="cri-o://537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e" gracePeriod=30 Oct 30 05:23:58.700845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.700819 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801502 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801627 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801700 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801766 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801877 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.801978 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.802033 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.802074 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.802183 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\") pod \"b97d1c81-21d0-45e5-b4ab-054306d87115\" (UID: \"b97d1c81-21d0-45e5-b4ab-054306d87115\") " Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.804094 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.804426 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.804641 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.805246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.804848 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.806069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.805687 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.809350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.809325 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:23:58.812526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.812484 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "kube-api-access-l459r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:58.813799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.813508 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:58.818126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.818079 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy" pod "b97d1c81-21d0-45e5-b4ab-054306d87115" (UID: "b97d1c81-21d0-45e5-b4ab-054306d87115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905026 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905053 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905063 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905072 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l459r\" (UniqueName: \"kubernetes.io/projected/b97d1c81-21d0-45e5-b4ab-054306d87115-kube-api-access-l459r\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905081 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905091 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905099 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905108 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97d1c81-21d0-45e5-b4ab-054306d87115-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.905509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:58.905116 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97d1c81-21d0-45e5-b4ab-054306d87115-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:23:59.402685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.402654 5058 generic.go:356] "Generic (PLEG): container finished" podID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerID="537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e" exitCode=0 Oct 30 05:23:59.402852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.402703 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" Oct 30 05:23:59.402852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.402729 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" event={"ID":"b97d1c81-21d0-45e5-b4ab-054306d87115","Type":"ContainerDied","Data":"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e"} Oct 30 05:23:59.402852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.402795 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l" event={"ID":"b97d1c81-21d0-45e5-b4ab-054306d87115","Type":"ContainerDied","Data":"cb919ea3d06b2ddc2df6ca5511377a40995e1c2096c8854e778b5f4dd8e8d0fd"} Oct 30 05:23:59.402852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.402819 5058 scope.go:122] "RemoveContainer" containerID="537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e" Oct 30 05:23:59.417430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.417395 5058 scope.go:122] "RemoveContainer" containerID="537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e" Oct 30 05:23:59.418033 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:23:59.418002 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e\": container with ID starting with 537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e not found: ID does not exist" containerID="537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e" Oct 30 05:23:59.418148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.418055 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e"} err="failed to get container status \"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e\": rpc error: code = NotFound desc = could not find container \"537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e\": container with ID starting with 537727e48a43925503c31788d5bf9a7786d1e2d6277567c1245ec6498065fd5e not found: ID does not exist" Oct 30 05:23:59.481808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.481723 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:23:59.499355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:23:59.499315 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l"] Oct 30 05:24:01.162424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:01.162389 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" path="/var/lib/kubelet/pods/b97d1c81-21d0-45e5-b4ab-054306d87115/volumes" Oct 30 05:24:03.759689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.759655 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:24:03.760635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760612 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" containerName="maas-api" Oct 30 05:24:03.760635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760635 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" containerName="maas-api" Oct 30 05:24:03.760803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760680 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" Oct 30 05:24:03.760803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760688 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" Oct 30 05:24:03.760928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760914 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6842bad2-fc0e-49fb-9e5f-649ff4ef60a2" containerName="maas-api" Oct 30 05:24:03.760982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.760937 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97d1c81-21d0-45e5-b4ab-054306d87115" containerName="istio-proxy" Oct 30 05:24:03.766232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.766201 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.785421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.785391 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34722e-openshift-default-dockercfg-nq64s\"" Oct 30 05:24:03.812425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812386 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812511 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812575 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812597 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812642 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812668 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.812852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812700 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nqg\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.813006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812850 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.813006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.812951 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.816319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.816291 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:24:03.914386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914343 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914491 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914561 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914585 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914642 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914666 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914694 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nqg\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914727 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.914977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.914770 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.915054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.915012 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.915117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.915027 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.915117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.915089 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.915584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.915559 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.915739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.915633 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.917554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.917522 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.917681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.917624 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.945147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.945108 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nqg\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:03.946402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:03.946369 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:04.084808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.084718 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:04.425777 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:24:04.425738 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b677a9_5b44_4b9f_a853_3d68e2ab3d94.slice/crio-7052433e2a7c7db50233417efb3e45abaef8909c8e5c840809e41a39cd87a40c WatchSource:0}: Error finding container 7052433e2a7c7db50233417efb3e45abaef8909c8e5c840809e41a39cd87a40c: Status 404 returned error can't find the container with id 7052433e2a7c7db50233417efb3e45abaef8909c8e5c840809e41a39cd87a40c Oct 30 05:24:04.426878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.426846 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:24:04.429112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.429077 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:04.429259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.429140 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:04.429259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.429170 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:04.454709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:04.454674 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" event={"ID":"54b677a9-5b44-4b9f-a853-3d68e2ab3d94","Type":"ContainerStarted","Data":"7052433e2a7c7db50233417efb3e45abaef8909c8e5c840809e41a39cd87a40c"} Oct 30 05:24:05.464744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:05.464708 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" event={"ID":"54b677a9-5b44-4b9f-a853-3d68e2ab3d94","Type":"ContainerStarted","Data":"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9"} Oct 30 05:24:05.537665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:05.537601 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podStartSLOduration=2.5375829579999998 podStartE2EDuration="2.537582958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:05.522549508 +0000 UTC m=+2418.492044497" watchObservedRunningTime="2026-10-30 05:24:05.537582958 +0000 UTC m=+2418.507077947" Oct 30 05:24:06.085298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:06.085265 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:06.086988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:06.086960 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:06.087124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:06.087016 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:06.669614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:06.666922 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:44188->10.132.0.191:15021: read: connection reset by peer" start-of-body= Oct 30 05:24:06.669614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:06.666982 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:44188->10.132.0.191:15021: read: connection reset by peer" Oct 30 05:24:07.085470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.085379 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:07.085470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.085443 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:07.481424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.481391 5058 generic.go:356] "Generic (PLEG): container finished" podID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerID="7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f" exitCode=0 Oct 30 05:24:07.481622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.481468 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerDied","Data":"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f"} Oct 30 05:24:07.482078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.482058 5058 scope.go:122] "RemoveContainer" containerID="7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f" Oct 30 05:24:07.483855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.483829 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:07.483917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.483898 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:07.483955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:07.483927 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:08.085742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.085701 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:08.086145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.085772 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:08.495892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.495853 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerStarted","Data":"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869"} Oct 30 05:24:08.578525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.578493 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:24:08.578708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.578650 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:08.578708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:08.578687 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:09.085882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:09.085835 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:09.086326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:09.085896 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:09.579241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:09.578828 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:09.579241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:09.578890 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:10.085126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:10.085097 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:10.085382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:10.085361 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:10.578794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:10.578758 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:10.579203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:10.578809 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:11.090696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:11.090659 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 05:24:11.091007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:11.090972 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 05:24:11.579010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:11.578969 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:11.579468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:11.579025 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:12.286660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:12.286628 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:12.287143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:12.287112 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:12.291861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:12.291837 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:24:12.579245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:12.579141 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:12.579245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:12.579231 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:13.578406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:13.578369 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 05:24:13.578605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:13.578424 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 05:24:14.583539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:14.583506 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:24:14.584010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:14.583996 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:24:14.584717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:14.584686 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:24:19.193074 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:24:19.193032 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:24:20.622485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:20.622452 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:24:25.623559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:25.623517 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" podUID="c5c4a1d0-7f84-4834-97f6-d890280f7257" containerName="payload-processing" containerID="cri-o://2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6" gracePeriod=25 Oct 30 05:24:25.907468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:25.907443 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:24:26.062584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.062551 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\") pod \"c5c4a1d0-7f84-4834-97f6-d890280f7257\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " Oct 30 05:24:26.062802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.062786 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\" (UniqueName: \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\") pod \"c5c4a1d0-7f84-4834-97f6-d890280f7257\" (UID: \"c5c4a1d0-7f84-4834-97f6-d890280f7257\") " Oct 30 05:24:26.063047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.063021 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume" pod "c5c4a1d0-7f84-4834-97f6-d890280f7257" (UID: "c5c4a1d0-7f84-4834-97f6-d890280f7257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:26.063353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.063327 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c4a1d0-7f84-4834-97f6-d890280f7257-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:24:26.064955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.064932 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v" pod "c5c4a1d0-7f84-4834-97f6-d890280f7257" (UID: "c5c4a1d0-7f84-4834-97f6-d890280f7257"). InnerVolumeSpecName "kube-api-access-dzz9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:26.164746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.164715 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzz9v\" (UniqueName: \"kubernetes.io/projected/c5c4a1d0-7f84-4834-97f6-d890280f7257-kube-api-access-dzz9v\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:24:26.669263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.669229 5058 generic.go:356] "Generic (PLEG): container finished" podID="c5c4a1d0-7f84-4834-97f6-d890280f7257" containerID="2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6" exitCode=0 Oct 30 05:24:26.669666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.669275 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" Oct 30 05:24:26.669666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.669310 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" event={"ID":"c5c4a1d0-7f84-4834-97f6-d890280f7257","Type":"ContainerDied","Data":"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6"} Oct 30 05:24:26.669666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.669365 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr" event={"ID":"c5c4a1d0-7f84-4834-97f6-d890280f7257","Type":"ContainerDied","Data":"f537d57a651fc2407e18a783c0492f6708598cb980777050dd227b978635d6f8"} Oct 30 05:24:26.669666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.669392 5058 scope.go:122] "RemoveContainer" containerID="2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6" Oct 30 05:24:26.681755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.681731 5058 scope.go:122] "RemoveContainer" containerID="2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6" Oct 30 05:24:26.682006 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:24:26.681987 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6\": container with ID starting with 2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6 not found: ID does not exist" containerID="2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6" Oct 30 05:24:26.682068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.682011 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6"} err="failed to get container status \"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6\": rpc error: code = NotFound desc = could not find container \"2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6\": container with ID starting with 2dbf7d3bf0da35fcd1afbe0fea10abdefd136499b55627c15f8a3de176c521a6 not found: ID does not exist" Oct 30 05:24:26.726402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.726371 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:24:26.739404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:26.739378 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr"] Oct 30 05:24:27.163447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:27.163416 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c4a1d0-7f84-4834-97f6-d890280f7257" path="/var/lib/kubelet/pods/c5c4a1d0-7f84-4834-97f6-d890280f7257/volumes" Oct 30 05:24:49.192314 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:24:49.192277 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:24:55.298705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:55.298668 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:24:55.299227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:55.298942 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" containerID="cri-o://516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9" gracePeriod=30 Oct 30 05:24:59.992852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:59.992766 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:24:59.993967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:59.993938 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c5c4a1d0-7f84-4834-97f6-d890280f7257" containerName="payload-processing" Oct 30 05:24:59.993967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:59.993965 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c4a1d0-7f84-4834-97f6-d890280f7257" containerName="payload-processing" Oct 30 05:24:59.994150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:59.994138 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c4a1d0-7f84-4834-97f6-d890280f7257" containerName="payload-processing" Oct 30 05:24:59.999697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:24:59.999672 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.018469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.018425 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:25:00.027495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.027470 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3855-dockercfg-7nq7g\"" Oct 30 05:25:00.027495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.027473 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3855\"" Oct 30 05:25:00.036571 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.036536 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.036720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.036665 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjr8\" (UniqueName: \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.138128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.138091 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.138328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.138171 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjr8\" (UniqueName: \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.138859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.138838 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.161739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.161705 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjr8\" (UniqueName: \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.318973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.318946 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:00.609342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.609318 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:25:00.627753 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:00.627591 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aa4bd2_7c5f_4ef4_b171_ff20b1c8b79f.slice/crio-8c86efa5c30e12089b912dc4e114687cef87f92d16a1bea3c9d4ef70b0098410 WatchSource:0}: Error finding container 8c86efa5c30e12089b912dc4e114687cef87f92d16a1bea3c9d4ef70b0098410: Status 404 returned error can't find the container with id 8c86efa5c30e12089b912dc4e114687cef87f92d16a1bea3c9d4ef70b0098410 Oct 30 05:25:00.630076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.630049 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:25:00.648482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648449 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.648617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648559 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.648689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648618 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.648750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648711 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:00.648750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648722 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.648865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648789 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.648923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.648879 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.649056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649025 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:00.649200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649142 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:00.649317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649263 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.649317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649302 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.649423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649337 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\") pod \"9b35bca4-8f53-433c-ab58-16a17c7a7831\" (UID: \"9b35bca4-8f53-433c-ab58-16a17c7a7831\") " Oct 30 05:25:00.649549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649522 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:00.649813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.649791 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:00.650415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.650394 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.650509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.650418 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.650509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.650431 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b35bca4-8f53-433c-ab58-16a17c7a7831-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.650509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.650443 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.650509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.650455 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.651388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.651362 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "kube-api-access-szqdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:00.651473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.651425 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:25:00.651473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.651432 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:00.651548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.651494 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy" pod "9b35bca4-8f53-433c-ab58-16a17c7a7831" (UID: "9b35bca4-8f53-433c-ab58-16a17c7a7831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:00.783839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.783795 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqdx\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-kube-api-access-szqdx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.783839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.783831 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.783839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.783844 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.784295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.783856 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b35bca4-8f53-433c-ab58-16a17c7a7831-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.982344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.982305 5058 generic.go:356] "Generic (PLEG): container finished" podID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerID="516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9" exitCode=0 Oct 30 05:25:00.982527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.982361 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" Oct 30 05:25:00.982527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.982394 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerDied","Data":"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9"} Oct 30 05:25:00.982527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.982450 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d" event={"ID":"9b35bca4-8f53-433c-ab58-16a17c7a7831","Type":"ContainerDied","Data":"eac2a8b3d066487da2dbc34c38731abb5475f234f9c5097c42c3fc2a734c9d50"} Oct 30 05:25:00.982527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.982474 5058 scope.go:122] "RemoveContainer" containerID="516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9" Oct 30 05:25:00.984041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.984017 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" event={"ID":"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f","Type":"ContainerStarted","Data":"8c86efa5c30e12089b912dc4e114687cef87f92d16a1bea3c9d4ef70b0098410"} Oct 30 05:25:00.994474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:00.994453 5058 scope.go:122] "RemoveContainer" containerID="637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e" Oct 30 05:25:01.006125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.006100 5058 scope.go:122] "RemoveContainer" containerID="516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9" Oct 30 05:25:01.006500 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:25:01.006474 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9\": container with ID starting with 516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9 not found: ID does not exist" containerID="516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9" Oct 30 05:25:01.006607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.006507 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9"} err="failed to get container status \"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9\": rpc error: code = NotFound desc = could not find container \"516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9\": container with ID starting with 516d9188d755a743e1b08cff75cdfb623aef9f50757f4678f4dd4b423ef0cdc9 not found: ID does not exist" Oct 30 05:25:01.006607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.006537 5058 scope.go:122] "RemoveContainer" containerID="637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e" Oct 30 05:25:01.006836 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:25:01.006814 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e\": container with ID starting with 637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e not found: ID does not exist" containerID="637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e" Oct 30 05:25:01.006935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.006840 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e"} err="failed to get container status \"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e\": rpc error: code = NotFound desc = could not find container \"637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e\": container with ID starting with 637c001a9ba8b55369f2b4851d2f9cfb4f5ddb736460d3ca4d067674a730cc1e not found: ID does not exist" Oct 30 05:25:01.051666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.051634 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:25:01.060101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.060069 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d"] Oct 30 05:25:01.164541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.164501 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" path="/var/lib/kubelet/pods/9b35bca4-8f53-433c-ab58-16a17c7a7831/volumes" Oct 30 05:25:01.250098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.250008 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:25:01.250871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.250849 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.250988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.250870 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.251109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.251094 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.251173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.251121 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.256273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.256253 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.273890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.273852 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-60a531\"" Oct 30 05:25:01.274187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.274163 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-60a531-dockercfg-c4jwm\"" Oct 30 05:25:01.276472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.276448 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:25:01.514290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.489314 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc5vq\" (UniqueName: \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.514290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.489610 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.514290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.500723 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:25:01.514290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.502006 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.514290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.502033 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b35bca4-8f53-433c-ab58-16a17c7a7831" containerName="istio-proxy" Oct 30 05:25:01.534201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.533589 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:25:01.534201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.533712 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.590927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.590890 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc5vq\" (UniqueName: \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.591130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.590974 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbcp\" (UniqueName: \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.591130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.591051 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.591362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.591305 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.592286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.592262 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.638957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.638918 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc5vq\" (UniqueName: \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\") pod \"payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.692457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.692408 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbcp\" (UniqueName: \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.692651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.692497 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.693642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.693614 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.716894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.716866 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbcp\" (UniqueName: \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\") pod \"payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:01.870614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.870534 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:01.882993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:01.882964 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:02.205514 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:02.205364 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58a5383_adad_49ef_809c_874b9c924121.slice/crio-5ea03138fc708b3318877269c6aea903ea470dd4d5d0b22a541dc57ec7e19f1e WatchSource:0}: Error finding container 5ea03138fc708b3318877269c6aea903ea470dd4d5d0b22a541dc57ec7e19f1e: Status 404 returned error can't find the container with id 5ea03138fc708b3318877269c6aea903ea470dd4d5d0b22a541dc57ec7e19f1e Oct 30 05:25:02.209593 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:02.209510 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99486445_a957_4d22_80ee_36257811cbb9.slice/crio-0b77b68b88fb42ef2b2f89052b4f9a72a9f7e5919b2d351533746911f443302f WatchSource:0}: Error finding container 0b77b68b88fb42ef2b2f89052b4f9a72a9f7e5919b2d351533746911f443302f: Status 404 returned error can't find the container with id 0b77b68b88fb42ef2b2f89052b4f9a72a9f7e5919b2d351533746911f443302f Oct 30 05:25:02.212043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:02.212020 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:25:02.218304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:02.218061 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:25:03.013588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.013550 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" event={"ID":"99486445-a957-4d22-80ee-36257811cbb9","Type":"ContainerStarted","Data":"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753"} Oct 30 05:25:03.013786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.013612 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" event={"ID":"99486445-a957-4d22-80ee-36257811cbb9","Type":"ContainerStarted","Data":"0b77b68b88fb42ef2b2f89052b4f9a72a9f7e5919b2d351533746911f443302f"} Oct 30 05:25:03.013786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.013644 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:03.015279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.015249 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" event={"ID":"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f","Type":"ContainerStarted","Data":"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98"} Oct 30 05:25:03.015407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.015387 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:03.016971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.016949 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" event={"ID":"d58a5383-adad-49ef-809c-874b9c924121","Type":"ContainerStarted","Data":"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463"} Oct 30 05:25:03.017069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.016987 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:03.017069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.017003 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" event={"ID":"d58a5383-adad-49ef-809c-874b9c924121","Type":"ContainerStarted","Data":"5ea03138fc708b3318877269c6aea903ea470dd4d5d0b22a541dc57ec7e19f1e"} Oct 30 05:25:03.075717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.075613 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" podStartSLOduration=2.075598007 podStartE2EDuration="2.075598007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:03.07293634 +0000 UTC m=+2476.042431329" watchObservedRunningTime="2026-10-30 05:25:03.075598007 +0000 UTC m=+2476.045093049" Oct 30 05:25:03.147518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.147457 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" podStartSLOduration=1.993619141 podStartE2EDuration="4.147437774s" totalImagesPullingTime="2.153818633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:03.142992802 +0000 UTC m=+2476.112487789" watchObservedRunningTime="2026-10-30 05:25:03.147437774 +0000 UTC m=+2476.116932762" Oct 30 05:25:03.194249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:03.194178 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" podStartSLOduration=2.19416486 podStartE2EDuration="2.19416486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:03.192442324 +0000 UTC m=+2476.161937311" watchObservedRunningTime="2026-10-30 05:25:03.19416486 +0000 UTC m=+2476.163659846" Oct 30 05:25:08.444490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.444450 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:25:08.451435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.451407 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.471870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.471839 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:25:08.481276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.481247 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dfb231\"/\"default-dockercfg-j5449\"" Oct 30 05:25:08.485785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.485755 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dfb231\"/\"openshift-service-ca.crt\"" Oct 30 05:25:08.485941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.485828 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dfb231\"/\"kube-root-ca.crt\"" Oct 30 05:25:08.486144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.486122 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dfb231\"/\"auth-test-model-dfb231-kserve-self-signed-certs\"" Oct 30 05:25:08.533349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533316 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.533546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533366 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.533546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533434 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.533677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533537 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.533677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533619 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.533677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.533647 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzd2\" (UniqueName: \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.634667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.634631 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.634883 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.634675 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzd2\" (UniqueName: \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.634964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.634949 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635012 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635050 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635086 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635344 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635393 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.635461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.635451 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.637035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.637007 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.637700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.637678 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.655920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.655894 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzd2\" (UniqueName: \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\") pod \"auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:08.768288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:08.767549 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:09.110658 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:09.110627 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c6ba582_05c4_4dd4_b7c5_98b0a2409bba.slice/crio-c569b0320a761dfb65353e05af40a5605b81d5641df0b83036500dfd7090cbd3 WatchSource:0}: Error finding container c569b0320a761dfb65353e05af40a5605b81d5641df0b83036500dfd7090cbd3: Status 404 returned error can't find the container with id c569b0320a761dfb65353e05af40a5605b81d5641df0b83036500dfd7090cbd3 Oct 30 05:25:09.111959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:09.111937 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:25:10.083328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:10.083293 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" event={"ID":"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba","Type":"ContainerStarted","Data":"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802"} Oct 30 05:25:10.083742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:10.083343 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" event={"ID":"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba","Type":"ContainerStarted","Data":"c569b0320a761dfb65353e05af40a5605b81d5641df0b83036500dfd7090cbd3"} Oct 30 05:25:10.156323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:10.155895 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" podStartSLOduration=2.155875046 podStartE2EDuration="2.155875046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:10.140248062 +0000 UTC m=+2483.109743051" watchObservedRunningTime="2026-10-30 05:25:10.155875046 +0000 UTC m=+2483.125370034" Oct 30 05:25:10.768316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:10.768284 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:14.028378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:14.028341 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:25:14.029163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:14.029144 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:25:14.029295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:14.029207 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:25:18.768654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:18.768616 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:18.793451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:18.793418 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:19.167333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:19.167299 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:25:19.193310 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:25:19.193276 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:25:31.088229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.088185 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:25:31.094625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.094601 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.114381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.114346 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:25:31.124260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.123604 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0dba31-openshift-default-dockercfg-9kc5b\"" Oct 30 05:25:31.186391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186355 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186388 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96dl\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186419 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186568 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186620 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186663 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186795 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186937 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.186984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.186964 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288162 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288335 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288386 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288414 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96dl\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288462 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288498 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288541 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288745 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288773 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.288850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288833 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.289015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.288899 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.289365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.289342 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.289479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.289379 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.290924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.290898 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.291030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.291014 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.309196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.309163 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96dl\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.309342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.309196 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.410823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.410793 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:31.723399 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:31.723369 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984f6054_dee9_4a26_a54c_e13180ed91b9.slice/crio-8a3f6375c91799f7e8541e5f181b81b63b790cfe5a19ddaabee25d2f363f18d2 WatchSource:0}: Error finding container 8a3f6375c91799f7e8541e5f181b81b63b790cfe5a19ddaabee25d2f363f18d2: Status 404 returned error can't find the container with id 8a3f6375c91799f7e8541e5f181b81b63b790cfe5a19ddaabee25d2f363f18d2 Oct 30 05:25:31.724905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.724876 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:25:31.726192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.726162 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:31.726301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.726240 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:31.726301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.726270 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:31.972518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.972447 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:25:31.977174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:31.977156 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.000978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.000949 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-262c10-openshift-default-dockercfg-gq5xt\"" Oct 30 05:25:32.001504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.001475 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:25:32.104838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.104794 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.104928 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105010 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105056 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105185 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105282 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4n2r\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105325 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105429 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.105668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.105451 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.212028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.211989 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.212237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.212054 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.212312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.212277 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.215941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.215906 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.215991 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.216051 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.216149 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.216251 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4n2r\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.216317 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.216929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.216903 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.217193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.217172 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.221295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.221262 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.221846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.221650 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.224405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.224344 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.228055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.228030 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.230657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.230630 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.238414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.238391 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.242547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.242524 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4n2r\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.259419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.259367 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" event={"ID":"984f6054-dee9-4a26-a54c-e13180ed91b9","Type":"ContainerStarted","Data":"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248"} Oct 30 05:25:32.259585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.259481 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" event={"ID":"984f6054-dee9-4a26-a54c-e13180ed91b9","Type":"ContainerStarted","Data":"8a3f6375c91799f7e8541e5f181b81b63b790cfe5a19ddaabee25d2f363f18d2"} Oct 30 05:25:32.292949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.292915 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:32.312660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.312601 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podStartSLOduration=1.312582604 podStartE2EDuration="1.312582604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:32.308764575 +0000 UTC m=+2505.278259562" watchObservedRunningTime="2026-10-30 05:25:32.312582604 +0000 UTC m=+2505.282077594" Oct 30 05:25:32.411592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.411557 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:32.413206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.413177 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:32.413326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.413252 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:32.633811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.633780 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:25:32.637520 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:32.637483 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6f7431_234b_4cda_85cb_6b2de9439561.slice/crio-c5f0c4890e61baa54d66d28b2a696612f46a1f368e478f439a418fb17430bf45 WatchSource:0}: Error finding container c5f0c4890e61baa54d66d28b2a696612f46a1f368e478f439a418fb17430bf45: Status 404 returned error can't find the container with id c5f0c4890e61baa54d66d28b2a696612f46a1f368e478f439a418fb17430bf45 Oct 30 05:25:32.640675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.640641 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:32.640793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.640711 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:32.640793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:32.640747 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:33.269369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.269332 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" event={"ID":"0b6f7431-234b-4cda-85cb-6b2de9439561","Type":"ContainerStarted","Data":"e45dee61b8405f0dc5bbd557468a4ef55925e7caa877e6d92068fef69bc93230"} Oct 30 05:25:33.269804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.269390 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" event={"ID":"0b6f7431-234b-4cda-85cb-6b2de9439561","Type":"ContainerStarted","Data":"c5f0c4890e61baa54d66d28b2a696612f46a1f368e478f439a418fb17430bf45"} Oct 30 05:25:33.293717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.293682 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:33.295181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.295148 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:33.295317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.295192 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:33.319196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.319133 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podStartSLOduration=2.319115175 podStartE2EDuration="2.319115175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:33.314273285 +0000 UTC m=+2506.283768270" watchObservedRunningTime="2026-10-30 05:25:33.319115175 +0000 UTC m=+2506.288610165" Oct 30 05:25:33.414128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.413660 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:33.414128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:33.413725 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:34.293273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:34.293237 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:34.293664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:34.293294 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:34.411116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:34.411080 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:34.411355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:34.411143 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:35.294006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:35.293973 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:35.294497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:35.294472 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:35.412046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:35.412015 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:35.412313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:35.412291 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:36.293306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:36.293271 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:36.293591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:36.293329 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:36.411718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:36.411679 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:36.412132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:36.411731 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:37.293719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:37.293680 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:37.293914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:37.293736 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:37.411320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:37.411286 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:37.411495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:37.411341 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:38.293870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:38.293833 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 05:25:38.294316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:38.293887 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 05:25:38.411529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:38.411496 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 05:25:38.411701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:38.411561 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 05:25:39.349541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.349509 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:39.351096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.350474 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:39.353770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.353741 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:25:39.416901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.416873 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:39.417166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.417147 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:39.417937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:39.417919 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:25:42.973446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:42.973404 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:25:42.981734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:42.981704 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:42.996327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:42.996287 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:25:43.012237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.004521 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-60a531-serving-cert\"" Oct 30 05:25:43.108075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.108034 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.108299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.108278 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdwt\" (UniqueName: \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.108391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.108375 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.108441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.108406 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.221301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.213651 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.221301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.213753 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.221301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.213995 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.221301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.214236 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdwt\" (UniqueName: \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.221301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.217113 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.248805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.248726 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdwt\" (UniqueName: \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.259302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.249781 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.264790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.264753 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\") pod \"maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.332242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.326899 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:43.713724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:43.713694 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:25:43.713984 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:25:43.713950 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d11cb1_651d_4dec_a4f4_0f7e98c34506.slice/crio-a6d719bef1a83c9e95ede9c96b6a547af0c58e2a66e49db1aa77d778dd651cb9 WatchSource:0}: Error finding container a6d719bef1a83c9e95ede9c96b6a547af0c58e2a66e49db1aa77d778dd651cb9: Status 404 returned error can't find the container with id a6d719bef1a83c9e95ede9c96b6a547af0c58e2a66e49db1aa77d778dd651cb9 Oct 30 05:25:44.438575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:44.438537 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" event={"ID":"a0d11cb1-651d-4dec-a4f4-0f7e98c34506","Type":"ContainerStarted","Data":"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29"} Oct 30 05:25:44.439044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:44.438591 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" event={"ID":"a0d11cb1-651d-4dec-a4f4-0f7e98c34506","Type":"ContainerStarted","Data":"a6d719bef1a83c9e95ede9c96b6a547af0c58e2a66e49db1aa77d778dd651cb9"} Oct 30 05:25:44.439044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:44.438636 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:44.486086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:44.486027 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" podStartSLOduration=2.115798404 podStartE2EDuration="2.486013968s" totalImagesPullingTime="370.215564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:44.484376625 +0000 UTC m=+2517.453871613" watchObservedRunningTime="2026-10-30 05:25:44.486013968 +0000 UTC m=+2517.455508955" Oct 30 05:25:46.572628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:46.572593 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:25:46.573114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:46.572861 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" podUID="789cc872-1991-457e-abf0-12e3eaea25c3" containerName="maas-api" containerID="cri-o://979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9" gracePeriod=30 Oct 30 05:25:46.969455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:46.969424 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:25:47.118977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.112682 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\") pod \"789cc872-1991-457e-abf0-12e3eaea25c3\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " Oct 30 05:25:47.118977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.112873 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\" (UniqueName: \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\") pod \"789cc872-1991-457e-abf0-12e3eaea25c3\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " Oct 30 05:25:47.118977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.113069 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\") pod \"789cc872-1991-457e-abf0-12e3eaea25c3\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " Oct 30 05:25:47.118977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.113129 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") pod \"789cc872-1991-457e-abf0-12e3eaea25c3\" (UID: \"789cc872-1991-457e-abf0-12e3eaea25c3\") " Oct 30 05:25:47.131631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.131594 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls" pod "789cc872-1991-457e-abf0-12e3eaea25c3" (UID: "789cc872-1991-457e-abf0-12e3eaea25c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:47.177824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.177723 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca" pod "789cc872-1991-457e-abf0-12e3eaea25c3" (UID: "789cc872-1991-457e-abf0-12e3eaea25c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:47.204410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.203712 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs" pod "789cc872-1991-457e-abf0-12e3eaea25c3" (UID: "789cc872-1991-457e-abf0-12e3eaea25c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:47.204410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.203938 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm" pod "789cc872-1991-457e-abf0-12e3eaea25c3" (UID: "789cc872-1991-457e-abf0-12e3eaea25c3"). InnerVolumeSpecName "kube-api-access-55lmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:47.215601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.215572 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789cc872-1991-457e-abf0-12e3eaea25c3-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:47.215601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.215600 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lmm\" (UniqueName: \"kubernetes.io/projected/789cc872-1991-457e-abf0-12e3eaea25c3-kube-api-access-55lmm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:47.215783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.215610 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:47.215783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.215619 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789cc872-1991-457e-abf0-12e3eaea25c3-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:25:47.461272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.461168 5058 generic.go:356] "Generic (PLEG): container finished" podID="789cc872-1991-457e-abf0-12e3eaea25c3" containerID="979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9" exitCode=0 Oct 30 05:25:47.461272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.461213 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" event={"ID":"789cc872-1991-457e-abf0-12e3eaea25c3","Type":"ContainerDied","Data":"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9"} Oct 30 05:25:47.461272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.461249 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" Oct 30 05:25:47.461560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.461295 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6" event={"ID":"789cc872-1991-457e-abf0-12e3eaea25c3","Type":"ContainerDied","Data":"64d75121c5c93fed51ac807a42789c2111b2a7e61422fa6c2d5d7f1d151ab565"} Oct 30 05:25:47.461560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.461334 5058 scope.go:122] "RemoveContainer" containerID="979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9" Oct 30 05:25:47.472345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.472323 5058 scope.go:122] "RemoveContainer" containerID="979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9" Oct 30 05:25:47.472592 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:25:47.472574 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9\": container with ID starting with 979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9 not found: ID does not exist" containerID="979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9" Oct 30 05:25:47.472641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.472601 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9"} err="failed to get container status \"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9\": rpc error: code = NotFound desc = could not find container \"979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9\": container with ID starting with 979abbfed9743b91a732023f107f73e515ae6f18b997c65990c31775680d0fb9 not found: ID does not exist" Oct 30 05:25:47.519245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.519198 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:25:47.532098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:47.532072 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-68b893-78f987b98b-tjhv6"] Oct 30 05:25:49.165291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:49.165209 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789cc872-1991-457e-abf0-12e3eaea25c3" path="/var/lib/kubelet/pods/789cc872-1991-457e-abf0-12e3eaea25c3/volumes" Oct 30 05:25:49.192899 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:25:49.192861 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:25:50.451302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:50.451275 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:25:54.866819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:54.866780 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:25:55.092629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:55.092592 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:25:55.092909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:25:55.092861 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" containerID="cri-o://60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86" gracePeriod=30 Oct 30 05:26:00.449183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.448748 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:26:00.534460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534425 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534530 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534561 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534664 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534691 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534688 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.534787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534722 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534900 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.534995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.534937 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.535108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535058 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\") pod \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\" (UID: \"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f\") " Oct 30 05:26:00.535342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535320 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:00.535729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535552 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.535729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535567 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.535729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535650 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.535945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535921 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.536001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535946 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.536001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535960 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.536001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535974 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.536001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.535988 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.539712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.539679 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:00.539810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.539739 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "kube-api-access-8cfgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:00.539919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.539897 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:26:00.540104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.540081 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy" pod "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" (UID: "cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.613741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.613705 5058 generic.go:356] "Generic (PLEG): container finished" podID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerID="60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86" exitCode=0 Oct 30 05:26:00.613944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.613754 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" Oct 30 05:26:00.613944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.613791 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" event={"ID":"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f","Type":"ContainerDied","Data":"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86"} Oct 30 05:26:00.613944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.613853 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt" event={"ID":"cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f","Type":"ContainerDied","Data":"412aacbd46ddea85dbfd42d5931b3333d1870e4a949b321b3af5c9f0faa4d27d"} Oct 30 05:26:00.613944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.613880 5058 scope.go:122] "RemoveContainer" containerID="60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86" Oct 30 05:26:00.625977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.625957 5058 scope.go:122] "RemoveContainer" containerID="60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86" Oct 30 05:26:00.626265 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:00.626239 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86\": container with ID starting with 60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86 not found: ID does not exist" containerID="60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86" Oct 30 05:26:00.626363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.626274 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86"} err="failed to get container status \"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86\": rpc error: code = NotFound desc = could not find container \"60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86\": container with ID starting with 60016409e843aa14e6d2ea9bb704aa3e49bc48460f48a11b7b0a887565671a86 not found: ID does not exist" Oct 30 05:26:00.636921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.636897 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.636921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.636920 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.637028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.636929 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.637028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.636937 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfgc\" (UniqueName: \"kubernetes.io/projected/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f-kube-api-access-8cfgc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.676318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.676252 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:26:00.681802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:00.681780 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt"] Oct 30 05:26:01.163879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:01.163843 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" path="/var/lib/kubelet/pods/cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f/volumes" Oct 30 05:26:09.889669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:09.889631 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" containerName="main" containerID="cri-o://651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799" gracePeriod=45 Oct 30 05:26:10.231433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.231408 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:26:10.366116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366085 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366148 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\" (UniqueName: \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366176 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366264 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366312 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366367 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:10.366574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366557 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\" (UniqueName: \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\") pod \"fee7355e-e53b-4365-90e0-66b341d0f577\" (UID: \"fee7355e-e53b-4365-90e0-66b341d0f577\") " Oct 30 05:26:10.366638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366559 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:10.366695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.366598 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:10.367140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.367119 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.367140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.367138 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.367364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.367146 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.368704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.368570 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:10.368704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.368660 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "kube-api-access-gm6rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:10.368704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.368674 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs" pod "fee7355e-e53b-4365-90e0-66b341d0f577" (UID: "fee7355e-e53b-4365-90e0-66b341d0f577"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:10.469127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.469057 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6rv\" (UniqueName: \"kubernetes.io/projected/fee7355e-e53b-4365-90e0-66b341d0f577-kube-api-access-gm6rv\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.469127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.469085 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee7355e-e53b-4365-90e0-66b341d0f577-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.469127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.469094 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee7355e-e53b-4365-90e0-66b341d0f577-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:10.725201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.725114 5058 generic.go:356] "Generic (PLEG): container finished" podID="fee7355e-e53b-4365-90e0-66b341d0f577" containerID="651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799" exitCode=0 Oct 30 05:26:10.725201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.725150 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" Oct 30 05:26:10.725461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.725196 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" event={"ID":"fee7355e-e53b-4365-90e0-66b341d0f577","Type":"ContainerDied","Data":"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799"} Oct 30 05:26:10.725461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.725257 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" event={"ID":"fee7355e-e53b-4365-90e0-66b341d0f577","Type":"ContainerDied","Data":"cd26d66d6dddd232d56e107f6dee494491f8e87026b9097c1f99b118a7c92c75"} Oct 30 05:26:10.725461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.725281 5058 scope.go:122] "RemoveContainer" containerID="651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799" Oct 30 05:26:10.737079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.737061 5058 scope.go:122] "RemoveContainer" containerID="651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799" Oct 30 05:26:10.737366 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:10.737346 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799\": container with ID starting with 651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799 not found: ID does not exist" containerID="651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799" Oct 30 05:26:10.737420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.737375 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799"} err="failed to get container status \"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799\": rpc error: code = NotFound desc = could not find container \"651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799\": container with ID starting with 651e90cefee54ce1c457c3b72d5670e3eeef587df7dac8d33e8c80569495f799 not found: ID does not exist" Oct 30 05:26:10.783697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.783666 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:26:10.787166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:10.787142 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r"] Oct 30 05:26:11.165172 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:11.165136 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" path="/var/lib/kubelet/pods/fee7355e-e53b-4365-90e0-66b341d0f577/volumes" Oct 30 05:26:11.171052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:11.171023 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-44bc69/sub-test-model-44bc69-kserve-b8479bd9c-tx76r" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:26:17.768657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:17.768619 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:26:19.192309 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:19.192275 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:26:31.328073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:31.328040 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:26:31.362180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:31.362150 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:26:32.791654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:32.791593 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" podUID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" containerName="main" containerID="cri-o://4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4" gracePeriod=45 Oct 30 05:26:33.074886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.074853 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:26:33.104317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.104286 5058 generic.go:356] "Generic (PLEG): container finished" podID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" containerID="4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4" exitCode=0 Oct 30 05:26:33.104482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.104326 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" Oct 30 05:26:33.104482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.104376 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" event={"ID":"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde","Type":"ContainerDied","Data":"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4"} Oct 30 05:26:33.104482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.104424 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz" event={"ID":"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde","Type":"ContainerDied","Data":"816f826f68e6408daddb09f5dc8ad9a0f06dd6769941c849de058fc98af1b5fa"} Oct 30 05:26:33.104482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.104446 5058 scope.go:122] "RemoveContainer" containerID="4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4" Oct 30 05:26:33.114275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.114259 5058 scope.go:122] "RemoveContainer" containerID="4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4" Oct 30 05:26:33.114565 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:33.114539 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4\": container with ID starting with 4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4 not found: ID does not exist" containerID="4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4" Oct 30 05:26:33.114621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.114573 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4"} err="failed to get container status \"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4\": rpc error: code = NotFound desc = could not find container \"4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4\": container with ID starting with 4dc2eb4a53041b2716b2d478b801aeab1c96c707f082a846f8c0dd44a8fa55f4 not found: ID does not exist" Oct 30 05:26:33.122113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122087 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122134 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122232 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122294 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122387 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122333 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:33.122499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122430 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122452 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:33.122499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122459 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\" (UniqueName: \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\") pod \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\" (UID: \"ec4a9e9f-9ebc-47b4-88ff-e581f2175fde\") " Oct 30 05:26:33.122822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.122798 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:33.123566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.123529 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.123720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.123698 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.123720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.123721 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.124840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.124816 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:33.124921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.124841 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:33.124961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.124937 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld" pod "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" (UID: "ec4a9e9f-9ebc-47b4-88ff-e581f2175fde"). InnerVolumeSpecName "kube-api-access-jr4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:33.225080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.225051 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.225080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.225076 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4ld\" (UniqueName: \"kubernetes.io/projected/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-kube-api-access-jr4ld\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.225080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.225086 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.448701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.448667 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:26:33.462284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:33.462256 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/sub-test-model-93bad9-kserve-55685849bf-cfpnz"] Oct 30 05:26:35.164058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:35.164025 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" path="/var/lib/kubelet/pods/ec4a9e9f-9ebc-47b4-88ff-e581f2175fde/volumes" Oct 30 05:26:36.329191 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.329154 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" podUID="99486445-a957-4d22-80ee-36257811cbb9" containerName="payload-processing" containerID="cri-o://ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753" gracePeriod=25 Oct 30 05:26:36.363060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.363030 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" podUID="d58a5383-adad-49ef-809c-874b9c924121" containerName="payload-pre-processing" containerID="cri-o://1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463" gracePeriod=25 Oct 30 05:26:36.654529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.654145 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:26:36.712594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.712568 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:26:36.778490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.778458 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\") pod \"d58a5383-adad-49ef-809c-874b9c924121\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " Oct 30 05:26:36.778688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.778600 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\" (UniqueName: \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\") pod \"d58a5383-adad-49ef-809c-874b9c924121\" (UID: \"d58a5383-adad-49ef-809c-874b9c924121\") " Oct 30 05:26:36.778805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.778787 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\") pod \"99486445-a957-4d22-80ee-36257811cbb9\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " Oct 30 05:26:36.778940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.778865 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\" (UniqueName: \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\") pod \"99486445-a957-4d22-80ee-36257811cbb9\" (UID: \"99486445-a957-4d22-80ee-36257811cbb9\") " Oct 30 05:26:36.778940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.778873 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume" pod "d58a5383-adad-49ef-809c-874b9c924121" (UID: "d58a5383-adad-49ef-809c-874b9c924121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:36.779331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.779310 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume" pod "99486445-a957-4d22-80ee-36257811cbb9" (UID: "99486445-a957-4d22-80ee-36257811cbb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:36.779487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.779466 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99486445-a957-4d22-80ee-36257811cbb9-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:36.779559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.779495 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58a5383-adad-49ef-809c-874b9c924121-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:36.780984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.780965 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp" pod "99486445-a957-4d22-80ee-36257811cbb9" (UID: "99486445-a957-4d22-80ee-36257811cbb9"). InnerVolumeSpecName "kube-api-access-8lbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:36.781307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.781287 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq" pod "d58a5383-adad-49ef-809c-874b9c924121" (UID: "d58a5383-adad-49ef-809c-874b9c924121"). InnerVolumeSpecName "kube-api-access-cc5vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:36.881187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.881095 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbcp\" (UniqueName: \"kubernetes.io/projected/99486445-a957-4d22-80ee-36257811cbb9-kube-api-access-8lbcp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:36.881187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:36.881125 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc5vq\" (UniqueName: \"kubernetes.io/projected/d58a5383-adad-49ef-809c-874b9c924121-kube-api-access-cc5vq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:26:37.138367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.138334 5058 generic.go:356] "Generic (PLEG): container finished" podID="d58a5383-adad-49ef-809c-874b9c924121" containerID="1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463" exitCode=0 Oct 30 05:26:37.138565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.138376 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" Oct 30 05:26:37.138565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.138416 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" event={"ID":"d58a5383-adad-49ef-809c-874b9c924121","Type":"ContainerDied","Data":"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463"} Oct 30 05:26:37.138565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.138472 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4" event={"ID":"d58a5383-adad-49ef-809c-874b9c924121","Type":"ContainerDied","Data":"5ea03138fc708b3318877269c6aea903ea470dd4d5d0b22a541dc57ec7e19f1e"} Oct 30 05:26:37.138565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.138502 5058 scope.go:122] "RemoveContainer" containerID="1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463" Oct 30 05:26:37.140414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.140388 5058 generic.go:356] "Generic (PLEG): container finished" podID="99486445-a957-4d22-80ee-36257811cbb9" containerID="ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753" exitCode=0 Oct 30 05:26:37.140553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.140460 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" event={"ID":"99486445-a957-4d22-80ee-36257811cbb9","Type":"ContainerDied","Data":"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753"} Oct 30 05:26:37.140553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.140504 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" event={"ID":"99486445-a957-4d22-80ee-36257811cbb9","Type":"ContainerDied","Data":"0b77b68b88fb42ef2b2f89052b4f9a72a9f7e5919b2d351533746911f443302f"} Oct 30 05:26:37.140553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.140551 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2" Oct 30 05:26:37.151657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.151637 5058 scope.go:122] "RemoveContainer" containerID="1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463" Oct 30 05:26:37.151936 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:37.151912 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463\": container with ID starting with 1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463 not found: ID does not exist" containerID="1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463" Oct 30 05:26:37.152020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.151942 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463"} err="failed to get container status \"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463\": rpc error: code = NotFound desc = could not find container \"1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463\": container with ID starting with 1b5b3ba48718d218f9c15d1d09108e1170d7b270d89ee6258b381234ff945463 not found: ID does not exist" Oct 30 05:26:37.152020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.151959 5058 scope.go:122] "RemoveContainer" containerID="ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753" Oct 30 05:26:37.163021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.163005 5058 scope.go:122] "RemoveContainer" containerID="ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753" Oct 30 05:26:37.163343 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:37.163313 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753\": container with ID starting with ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753 not found: ID does not exist" containerID="ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753" Oct 30 05:26:37.163440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.163351 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753"} err="failed to get container status \"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753\": rpc error: code = NotFound desc = could not find container \"ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753\": container with ID starting with ab7d8402e73336493fb2370d88fe5d0df38927ca241d5cbdf2e30a0539fd2753 not found: ID does not exist" Oct 30 05:26:37.200014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.199988 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:26:37.213440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.213417 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4"] Oct 30 05:26:37.258462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.258427 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:26:37.264927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:37.264890 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2"] Oct 30 05:26:39.201530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:39.201472 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99486445-a957-4d22-80ee-36257811cbb9" path="/var/lib/kubelet/pods/99486445-a957-4d22-80ee-36257811cbb9/volumes" Oct 30 05:26:39.202346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:39.202317 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58a5383-adad-49ef-809c-874b9c924121" path="/var/lib/kubelet/pods/d58a5383-adad-49ef-809c-874b9c924121/volumes" Oct 30 05:26:42.659792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.659757 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:26:42.660562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660544 5058 cpu_manager.go:414] "containerMap: removing container" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660564 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660576 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" containerName="main" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660581 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" containerName="main" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660596 5058 cpu_manager.go:414] "containerMap: removing container" podUID="99486445-a957-4d22-80ee-36257811cbb9" containerName="payload-processing" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660600 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99486445-a957-4d22-80ee-36257811cbb9" containerName="payload-processing" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660608 5058 cpu_manager.go:414] "containerMap: removing container" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" containerName="main" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660613 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" containerName="main" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660619 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d58a5383-adad-49ef-809c-874b9c924121" containerName="payload-pre-processing" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660624 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58a5383-adad-49ef-809c-874b9c924121" containerName="payload-pre-processing" Oct 30 05:26:42.660632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660635 5058 cpu_manager.go:414] "containerMap: removing container" podUID="789cc872-1991-457e-abf0-12e3eaea25c3" containerName="maas-api" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660640 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789cc872-1991-457e-abf0-12e3eaea25c3" containerName="maas-api" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660753 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb50fd7b-3c72-47f6-8a9c-cb9c52f5747f" containerName="istio-proxy" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660764 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4a9e9f-9ebc-47b4-88ff-e581f2175fde" containerName="main" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660772 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="99486445-a957-4d22-80ee-36257811cbb9" containerName="payload-processing" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660778 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee7355e-e53b-4365-90e0-66b341d0f577" containerName="main" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660783 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58a5383-adad-49ef-809c-874b9c924121" containerName="payload-pre-processing" Oct 30 05:26:42.660965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.660790 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="789cc872-1991-457e-abf0-12e3eaea25c3" containerName="maas-api" Oct 30 05:26:42.666115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.666096 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.686938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.686905 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:26:42.694454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.694428 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-35a94e-serving-cert\"" Oct 30 05:26:42.767565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.767529 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.767721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.767572 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.767721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.767591 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.767870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.767852 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbt6q\" (UniqueName: \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.886565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.875410 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbt6q\" (UniqueName: \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.886565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.875564 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.886565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.875606 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.886565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.875639 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.886565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.877067 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.890487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.890356 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.915943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.915869 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.916846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.916816 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbt6q\" (UniqueName: \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\") pod \"maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:42.980950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:42.980919 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:43.285710 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:26:43.285681 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9f62ec_4c65_45e0_8f4e_725f14779dfb.slice/crio-1ef937edb982193353a3571738cb6da10db16da07dc51eb9570b5c7b9a30fb7a WatchSource:0}: Error finding container 1ef937edb982193353a3571738cb6da10db16da07dc51eb9570b5c7b9a30fb7a: Status 404 returned error can't find the container with id 1ef937edb982193353a3571738cb6da10db16da07dc51eb9570b5c7b9a30fb7a Oct 30 05:26:43.286175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:43.286150 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:26:44.232489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:44.232456 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" event={"ID":"2b9f62ec-4c65-45e0-8f4e-725f14779dfb","Type":"ContainerStarted","Data":"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0"} Oct 30 05:26:44.232895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:44.232504 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" event={"ID":"2b9f62ec-4c65-45e0-8f4e-725f14779dfb","Type":"ContainerStarted","Data":"1ef937edb982193353a3571738cb6da10db16da07dc51eb9570b5c7b9a30fb7a"} Oct 30 05:26:44.232895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:44.232592 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:44.302529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:44.302464 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" podStartSLOduration=1.9820291939999999 podStartE2EDuration="2.302442936s" totalImagesPullingTime="320.413742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:44.286151372 +0000 UTC m=+2577.255646362" watchObservedRunningTime="2026-10-30 05:26:44.302442936 +0000 UTC m=+2577.271937926" Oct 30 05:26:48.783401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.783368 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:26:48.788588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.788563 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.809669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.809632 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:26:48.816517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.816478 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"openshift-service-ca.crt\"" Oct 30 05:26:48.816517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.816489 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"kube-root-ca.crt\"" Oct 30 05:26:48.819107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.819084 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"rate-test-model-44bc69-kserve-self-signed-certs\"" Oct 30 05:26:48.819107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.819089 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-44bc69\"/\"default-dockercfg-4vp7n\"" Oct 30 05:26:48.885841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.885807 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.886022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.885851 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.886075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.886037 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2bc\" (UniqueName: \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.886150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.886132 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.886195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.886162 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.886195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.886178 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987487 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987522 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987602 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2bc\" (UniqueName: \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987683 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987707 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987733 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987888 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.987981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.987942 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.988072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.988043 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.990003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.989978 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:48.990268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:48.990254 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:49.015269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:49.015242 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2bc\" (UniqueName: \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\") pod \"rate-test-model-44bc69-kserve-c59f6549-6mb2k\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:49.114606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:49.114517 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:49.228949 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:26:49.228908 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:26:49.495632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:49.495597 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:26:49.496128 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:26:49.496099 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7181c02_1b18_406a_bae2_a8323b5997c4.slice/crio-3c23cd7e84edece034187ba841b391bc09151276e32c00f591f32647c0757e45 WatchSource:0}: Error finding container 3c23cd7e84edece034187ba841b391bc09151276e32c00f591f32647c0757e45: Status 404 returned error can't find the container with id 3c23cd7e84edece034187ba841b391bc09151276e32c00f591f32647c0757e45 Oct 30 05:26:50.245442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:50.245415 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:26:50.290939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:50.290901 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" event={"ID":"b7181c02-1b18-406a-bae2-a8323b5997c4","Type":"ContainerStarted","Data":"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2"} Oct 30 05:26:50.291121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:50.290949 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" event={"ID":"b7181c02-1b18-406a-bae2-a8323b5997c4","Type":"ContainerStarted","Data":"3c23cd7e84edece034187ba841b391bc09151276e32c00f591f32647c0757e45"} Oct 30 05:26:50.338441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:50.338381 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" podStartSLOduration=2.338363128 podStartE2EDuration="2.338363128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:50.33579779 +0000 UTC m=+2583.305292791" watchObservedRunningTime="2026-10-30 05:26:50.338363128 +0000 UTC m=+2583.307858119" Oct 30 05:26:51.115380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:51.115349 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:59.116535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:59.116495 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:59.128596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:59.128570 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:26:59.373738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:26:59.373668 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:27:12.832314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.832274 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:27:12.876135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.876098 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:27:12.877660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.877636 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:12.908568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.908538 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-60a531-dockercfg-8rsmg\"" Oct 30 05:27:12.918012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.916875 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-60a531\"" Oct 30 05:27:12.973432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.973394 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:12.973432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:12.973438 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzrg\" (UniqueName: \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.075199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.075162 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.075199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.075203 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzrg\" (UniqueName: \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.075916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.075898 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.096972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.096904 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzrg\" (UniqueName: \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\") pod \"payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.203399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.203370 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:13.501673 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:27:13.501639 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b3028b_d4ca_4704_baa7_778e70a0ccdd.slice/crio-6da6d8f119b82a82f1ef4b0866dd0813ceb83d6422793698b18837c25d9efb9c WatchSource:0}: Error finding container 6da6d8f119b82a82f1ef4b0866dd0813ceb83d6422793698b18837c25d9efb9c: Status 404 returned error can't find the container with id 6da6d8f119b82a82f1ef4b0866dd0813ceb83d6422793698b18837c25d9efb9c Oct 30 05:27:13.503496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.503476 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:27:13.591203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:13.591167 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" event={"ID":"71b3028b-d4ca-4704-baa7-778e70a0ccdd","Type":"ContainerStarted","Data":"6da6d8f119b82a82f1ef4b0866dd0813ceb83d6422793698b18837c25d9efb9c"} Oct 30 05:27:14.599624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:14.599586 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" event={"ID":"71b3028b-d4ca-4704-baa7-778e70a0ccdd","Type":"ContainerStarted","Data":"093e970481a98d6101013f05c80356ef3c62d4582b946e381ad6d4cacdac0937"} Oct 30 05:27:14.600022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:14.599656 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:14.647283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:14.647235 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" podStartSLOduration=2.647209094 podStartE2EDuration="2.647209094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:14.644071008 +0000 UTC m=+2607.613566009" watchObservedRunningTime="2026-10-30 05:27:14.647209094 +0000 UTC m=+2607.616704082" Oct 30 05:27:19.150253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.150188 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:27:19.155345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.155318 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.171763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.171739 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45607d-openshift-default-dockercfg-dlgvx\"" Oct 30 05:27:19.192117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.192085 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:27:19.207332 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:27:19.205314 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:27:19.273473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273438 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273508 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273646 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwjz\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273710 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273733 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273799 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273888 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273927 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.273973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.273954 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.375780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.375744 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.375939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.375900 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.375996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.375942 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.375996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.375972 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376039 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376064 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376117 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwjz\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376179 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376200 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376211 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376565 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376637 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376721 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.376867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.376786 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.378342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.378318 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.378589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.378573 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.397548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.397520 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.397672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.397656 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwjz\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.474119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.474033 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:19.789702 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:27:19.789671 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6622fff6_ca8a_4ed5_869b_12f291249c2e.slice/crio-950074fb648674ff2264d58dc5cac0c8a399a8140aed640745a8d4215da83b6c WatchSource:0}: Error finding container 950074fb648674ff2264d58dc5cac0c8a399a8140aed640745a8d4215da83b6c: Status 404 returned error can't find the container with id 950074fb648674ff2264d58dc5cac0c8a399a8140aed640745a8d4215da83b6c Oct 30 05:27:19.790573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.790546 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:27:19.792516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.792486 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:19.792586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.792549 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:19.792586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:19.792581 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:20.656165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:20.656122 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" event={"ID":"6622fff6-ca8a-4ed5-869b-12f291249c2e","Type":"ContainerStarted","Data":"4be7209799743dbfadd969b62071c5c358f8111b3a50117f456d5087718a9842"} Oct 30 05:27:20.656798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:20.656248 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" event={"ID":"6622fff6-ca8a-4ed5-869b-12f291249c2e","Type":"ContainerStarted","Data":"950074fb648674ff2264d58dc5cac0c8a399a8140aed640745a8d4215da83b6c"} Oct 30 05:27:20.731976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:20.731913 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podStartSLOduration=1.731899101 podStartE2EDuration="1.731899101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:20.730653709 +0000 UTC m=+2613.700148696" watchObservedRunningTime="2026-10-30 05:27:20.731899101 +0000 UTC m=+2613.701394088" Oct 30 05:27:21.474760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:21.474722 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:21.476666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:21.476644 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 05:27:21.476781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:21.476710 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 05:27:22.475154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:22.475119 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 05:27:22.475592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:22.475184 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 05:27:23.475203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:23.475168 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 05:27:23.475637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:23.475266 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 05:27:24.475112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:24.475075 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 05:27:24.475347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:24.475129 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 05:27:25.474549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:25.474504 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 05:27:25.474776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:25.474572 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 05:27:25.608890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:25.608854 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:27:26.479435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.479400 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:26.479914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.479895 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:26.480665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.480628 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:27:26.717759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.717725 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:27:26.724458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.724431 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.802889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.802857 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-35a94e-openshift-default-dockercfg-s9msl\"" Oct 30 05:27:26.803937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.803909 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:27:26.901231 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901183 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901262 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901320 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901409 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901464 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901485 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901585 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz52v\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901893 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901876 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:26.901982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:26.901959 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003373 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003418 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003448 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003438 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003470 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003518 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003546 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz52v\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003832 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003857 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003879 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003914 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.003970 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.003938 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.004299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.004279 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.004498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.004481 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.006130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.006100 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.006292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.006107 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.026648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.026619 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.026806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.026787 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz52v\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.110125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.110094 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:27.427289 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:27:27.427258 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2d872c_1423_440f_8e42_0bd43c2c4f07.slice/crio-2073fb7ad2c06ea0ad5e797d09e10d3e2e84ddee9f2caaaceeeb718be9fbefa0 WatchSource:0}: Error finding container 2073fb7ad2c06ea0ad5e797d09e10d3e2e84ddee9f2caaaceeeb718be9fbefa0: Status 404 returned error can't find the container with id 2073fb7ad2c06ea0ad5e797d09e10d3e2e84ddee9f2caaaceeeb718be9fbefa0 Oct 30 05:27:27.428832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.428806 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:27:27.430183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.430150 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.430285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.430229 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.430285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.430259 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.716904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.716815 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" event={"ID":"db2d872c-1423-440f-8e42-0bd43c2c4f07","Type":"ContainerStarted","Data":"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3"} Oct 30 05:27:27.716904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.716874 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" event={"ID":"db2d872c-1423-440f-8e42-0bd43c2c4f07","Type":"ContainerStarted","Data":"2073fb7ad2c06ea0ad5e797d09e10d3e2e84ddee9f2caaaceeeb718be9fbefa0"} Oct 30 05:27:27.767051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:27.766996 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podStartSLOduration=1.766981556 podStartE2EDuration="1.766981556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:27.763826155 +0000 UTC m=+2620.733321145" watchObservedRunningTime="2026-10-30 05:27:27.766981556 +0000 UTC m=+2620.736476543" Oct 30 05:27:28.110886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:28.110797 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:28.112826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:28.112794 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:28.112944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:28.112842 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:29.113559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.113516 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:29.114051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.113621 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:29.718959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.718921 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:27:29.762820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.762790 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:29.769644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.769408 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:27:29.784407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.784328 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ec92c7-openshift-default-dockercfg-4sfsh\"" Oct 30 05:27:29.984610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:29.984514 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.006243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.006191 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.006754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.006726 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.007491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.007469 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vfc\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.007734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.007715 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.035696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.035654 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.038585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.038553 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.067810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.067771 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.068105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.068088 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.113672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.113634 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:30.114321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.114291 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170081 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170168 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170241 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170270 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170308 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170442 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170522 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vfc\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.170609 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.171103 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.181249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.172086 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.194247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.183839 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.194247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.185428 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.194247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.185691 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.238672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.238546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.241205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.239433 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.253887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.253859 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.263254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.259281 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vfc\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\") pod \"e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.404284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.401658 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:30.776807 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:27:30.776779 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914d0f0a_6044_4318_ad23_338d7e553377.slice/crio-68f8b014a3890c61cce968ea30d97c11350c75f049aee6a96bb3d97857313d95 WatchSource:0}: Error finding container 68f8b014a3890c61cce968ea30d97c11350c75f049aee6a96bb3d97857313d95: Status 404 returned error can't find the container with id 68f8b014a3890c61cce968ea30d97c11350c75f049aee6a96bb3d97857313d95 Oct 30 05:27:30.778278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.778255 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:27:30.780146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.780110 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:30.780360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.780189 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:30.780360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:30.780268 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:31.110506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:31.110422 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:31.110506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:31.110478 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:31.792070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:31.792035 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerStarted","Data":"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c"} Oct 30 05:27:31.792492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:31.792093 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerStarted","Data":"68f8b014a3890c61cce968ea30d97c11350c75f049aee6a96bb3d97857313d95"} Oct 30 05:27:31.849280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:31.847616 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podStartSLOduration=2.847597442 podStartE2EDuration="2.847597442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:31.844725241 +0000 UTC m=+2624.814220243" watchObservedRunningTime="2026-10-30 05:27:31.847597442 +0000 UTC m=+2624.817092430" Oct 30 05:27:32.111374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.111287 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:32.111374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.111350 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:32.402139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.402108 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:32.408392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.408368 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:32.807628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.807548 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:32.808639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:32.808616 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:27:33.111099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:33.111005 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 05:27:33.111099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:33.111066 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 05:27:34.366866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:34.366832 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:34.369097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:34.369073 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:34.370154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:34.370130 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:27:49.192495 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:27:49.192454 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:27:49.326351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.326319 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:27:49.331784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.331762 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.349017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.348993 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:27:49.363371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.363348 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.373723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.373692 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:27:49.387578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.387549 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:27:49.412979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.412947 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413028 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413102 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413149 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413280 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413318 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q6b\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413408 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413433 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413453 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413562 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkp5g\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413589 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413604 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413761 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.413854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413805 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413866 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413886 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.413985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.413904 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.515633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515545 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.515633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515614 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515660 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515689 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515759 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515783 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q6b\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515852 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515881 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.515915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515912 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515958 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.515995 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkp5g\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516041 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516067 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516157 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516197 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516262 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516262 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516331 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516365 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516366 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516519 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516628 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.516905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516833 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.516905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.516902 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.517212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.517020 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.517212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.517208 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.517390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.517295 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.517443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.517415 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.518629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.518605 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.518734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.518641 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.518734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.518669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.519562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.519540 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.540601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.540571 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.540769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.540746 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.540849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.540835 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q6b\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.541028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.541010 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkp5g\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.653900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.653863 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:49.688697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:49.688663 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:49.997775 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:27:49.997748 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52de5999_f679_4863_ae87_ca4d9d7059d1.slice/crio-e96a7849fd21853da96fc3794ee05c3ca587fadf518f2bf75c72cf7040ccae53 WatchSource:0}: Error finding container e96a7849fd21853da96fc3794ee05c3ca587fadf518f2bf75c72cf7040ccae53: Status 404 returned error can't find the container with id e96a7849fd21853da96fc3794ee05c3ca587fadf518f2bf75c72cf7040ccae53 Oct 30 05:27:50.000981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.000950 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:27:50.137353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.136349 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.137556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.137431 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.137556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.137533 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.168553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.168401 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:27:50.201785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.201246 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.201785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.201338 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.201785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.201374 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.980325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.980287 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" event={"ID":"52de5999-f679-4863-ae87-ca4d9d7059d1","Type":"ContainerStarted","Data":"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9"} Oct 30 05:27:50.980527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.980358 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" event={"ID":"52de5999-f679-4863-ae87-ca4d9d7059d1","Type":"ContainerStarted","Data":"e96a7849fd21853da96fc3794ee05c3ca587fadf518f2bf75c72cf7040ccae53"} Oct 30 05:27:50.982131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.982106 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" event={"ID":"d653fdb2-a81a-429f-838d-538ea3e59a11","Type":"ContainerStarted","Data":"7fe61d751c20aee10c653e6fccb9db6cac4715cfcb6ad25403cdafd1fbddcaa1"} Oct 30 05:27:50.982265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:50.982154 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" event={"ID":"d653fdb2-a81a-429f-838d-538ea3e59a11","Type":"ContainerStarted","Data":"e6dd5e686f05e0291b63f302ed412452e3bac73c197e66664e250427a59cbc01"} Oct 30 05:27:51.031829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.031779 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podStartSLOduration=2.031764801 podStartE2EDuration="2.031764801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:51.027625182 +0000 UTC m=+2643.997120170" watchObservedRunningTime="2026-10-30 05:27:51.031764801 +0000 UTC m=+2644.001259789" Oct 30 05:27:51.081618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.081567 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podStartSLOduration=2.081552811 podStartE2EDuration="2.081552811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:51.076616012 +0000 UTC m=+2644.046111025" watchObservedRunningTime="2026-10-30 05:27:51.081552811 +0000 UTC m=+2644.051047799" Oct 30 05:27:51.661242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.656583 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:51.661778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.661330 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 05:27:51.661778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.661389 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 05:27:51.689095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.689059 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:51.690535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.690508 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:51.690667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:51.690555 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:52.654677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:52.654635 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 05:27:52.654984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:52.654953 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 05:27:52.689396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:52.689362 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:52.689799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:52.689419 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:53.660244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:53.654620 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 05:27:53.660244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:53.654684 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 05:27:53.689732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:53.689690 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:53.690129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:53.689743 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:54.658242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:54.657698 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 05:27:54.658242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:54.657776 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 05:27:54.689181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:54.689136 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:54.689386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:54.689206 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:55.657241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:55.656884 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 05:27:55.657241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:55.656984 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 05:27:55.690375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:55.690335 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:55.690570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:55.690417 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:56.689260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:56.689203 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 05:27:56.689917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:56.689878 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 05:27:56.728682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:56.728546 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:56.729290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:56.729261 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:56.730166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:56.730142 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:27:57.694622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:57.694587 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:57.695078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:57.695061 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:27:57.695769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:27:57.695749 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:28:01.020488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:01.020450 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:28:04.395896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.395856 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:28:04.624466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.624434 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.671352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.628691 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:28:04.807587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.807548 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.808545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.808382 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.808842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.808823 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.811570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.811548 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.811839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.811820 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.824295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.823499 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjm5n\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.824295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.823555 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.824295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.823640 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.824295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.823701 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925337 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925392 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjm5n\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925432 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925488 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925544 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925606 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925719 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.925833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925780 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.926132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.925898 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.926495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.926465 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.927114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.927091 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.927677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.927651 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.928510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.928486 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.928854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.928829 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.933266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.933178 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.935267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.935244 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.949689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.949663 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:04.949867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:04.949842 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjm5n\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:05.026125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.026081 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:05.382041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.381624 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:28:05.437253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.431537 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:05.437253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.431637 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:05.437253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.431681 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:05.472256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:05.472184 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" event={"ID":"5d11365c-9280-4ea1-857e-218e2a1f46af","Type":"ContainerStarted","Data":"1fcc87fa55e32e67b70cde33f1ceef3134516e5e9673aa1821993c67e3483f47"} Oct 30 05:28:06.481630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:06.481587 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" event={"ID":"5d11365c-9280-4ea1-857e-218e2a1f46af","Type":"ContainerStarted","Data":"56b82a7ab1b84569b56bf35f001b7dadd3100335dc2aa45b710ad000fd0dcda7"} Oct 30 05:28:06.557057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:06.556987 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podStartSLOduration=2.556963908 podStartE2EDuration="2.556963908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:06.544723555 +0000 UTC m=+2659.514218556" watchObservedRunningTime="2026-10-30 05:28:06.556963908 +0000 UTC m=+2659.526458896" Oct 30 05:28:07.030996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:07.030960 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:07.031329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:07.031307 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:28:07.031476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:07.031452 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 05:28:08.028608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:08.028570 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:28:08.029187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:08.029156 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 05:28:09.030628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:09.030591 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:28:09.031241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:09.031185 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 05:28:10.046243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:10.045042 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:28:10.046243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:10.045115 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 05:28:11.035265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.028600 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:28:11.035265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.028672 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 05:28:11.908241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.896717 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:28:11.927867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.927828 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.933635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.933594 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:28:11.990283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990209 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990460 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990373 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990508 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990662 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990753 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990740 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990789 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhr9\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990849 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990904 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:11.990988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:11.990959 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.032198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.032168 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:12.032656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.032635 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:12.033378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.033357 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:28:12.092041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092005 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092083 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092175 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092371 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092456 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092484 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhr9\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092516 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092518 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092592 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092610 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092700 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.092858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092832 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.093012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.092996 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.093263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.093237 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.094779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.094759 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.094936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.094920 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.114680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.114655 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.114802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.114784 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhr9\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.254485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.254384 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:12.584616 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:28:12.584585 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb885da70_8b9f_47a5_933d_ad47e65bd220.slice/crio-184d87f07205124796441c292e2dfb9faa3d969b9a80be8c5a8f4b237b045458 WatchSource:0}: Error finding container 184d87f07205124796441c292e2dfb9faa3d969b9a80be8c5a8f4b237b045458: Status 404 returned error can't find the container with id 184d87f07205124796441c292e2dfb9faa3d969b9a80be8c5a8f4b237b045458 Oct 30 05:28:12.586722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.586697 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:28:12.587649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.587615 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.587752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.587692 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.587752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.587739 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.685468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.685432 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" event={"ID":"b885da70-8b9f-47a5-933d-ad47e65bd220","Type":"ContainerStarted","Data":"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d"} Oct 30 05:28:12.685652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.685506 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" event={"ID":"b885da70-8b9f-47a5-933d-ad47e65bd220","Type":"ContainerStarted","Data":"184d87f07205124796441c292e2dfb9faa3d969b9a80be8c5a8f4b237b045458"} Oct 30 05:28:12.747332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:12.747278 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podStartSLOduration=1.7472628019999998 podStartE2EDuration="1.747262802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:12.743038032 +0000 UTC m=+2665.712533020" watchObservedRunningTime="2026-10-30 05:28:12.747262802 +0000 UTC m=+2665.716757793" Oct 30 05:28:13.255650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:13.255614 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:13.257260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:13.257231 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:13.257427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:13.257281 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:14.255925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:14.255882 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:14.256431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:14.255935 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:15.256290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:15.256208 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:15.256691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:15.256309 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:16.050309 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.050262 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerName="main" containerID="cri-o://6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802" gracePeriod=45 Oct 30 05:28:16.156740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.156697 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": dial tcp 10.132.0.202:8000: connect: connection refused" Oct 30 05:28:16.256701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.256661 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:16.257154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.256724 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:16.379902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.379876 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:28:16.494042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494004 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494067 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494160 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\" (UniqueName: \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494212 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494385 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494450 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\") pod \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\" (UID: \"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba\") " Oct 30 05:28:16.494795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494766 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:16.494983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.494956 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:16.495051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.495019 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:16.495356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.495333 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.495447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.495363 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.495447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.495389 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.497386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.497365 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:16.499842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.499818 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "kube-api-access-xnzd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:16.499934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.499919 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs" pod "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" (UID: "7c6ba582-05c4-4dd4-b7c5-98b0a2409bba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:16.596420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.596389 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.596420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.596419 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.596626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.596432 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzd2\" (UniqueName: \"kubernetes.io/projected/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba-kube-api-access-xnzd2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:16.764439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.764404 5058 generic.go:356] "Generic (PLEG): container finished" podID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerID="6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802" exitCode=0 Oct 30 05:28:16.764621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.764472 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" event={"ID":"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba","Type":"ContainerDied","Data":"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802"} Oct 30 05:28:16.764621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.764498 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" event={"ID":"7c6ba582-05c4-4dd4-b7c5-98b0a2409bba","Type":"ContainerDied","Data":"c569b0320a761dfb65353e05af40a5605b81d5641df0b83036500dfd7090cbd3"} Oct 30 05:28:16.764621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.764514 5058 scope.go:122] "RemoveContainer" containerID="6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802" Oct 30 05:28:16.764740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.764631 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7" Oct 30 05:28:16.779785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.779749 5058 scope.go:122] "RemoveContainer" containerID="6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802" Oct 30 05:28:16.780043 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:16.780027 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802\": container with ID starting with 6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802 not found: ID does not exist" containerID="6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802" Oct 30 05:28:16.780102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.780049 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802"} err="failed to get container status \"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802\": rpc error: code = NotFound desc = could not find container \"6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802\": container with ID starting with 6ff475a754610fd1809bb0603b5af9c979606a8a8641f807d59b88815b264802 not found: ID does not exist" Oct 30 05:28:16.849333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.849261 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:28:16.853647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:16.853615 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dfb231/auth-test-model-dfb231-kserve-b7ff98dfc-7t9c7"] Oct 30 05:28:17.164444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:17.164408 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" path="/var/lib/kubelet/pods/7c6ba582-05c4-4dd4-b7c5-98b0a2409bba/volumes" Oct 30 05:28:17.257160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:17.256664 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:17.257160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:17.256739 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:18.255915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:18.255879 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 05:28:18.256127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:18.255935 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 05:28:18.852797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:18.852762 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:28:18.853201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:18.852981 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" containerID="cri-o://a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759" gracePeriod=30 Oct 30 05:28:19.018446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.018410 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:28:19.019379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.019357 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerName="main" Oct 30 05:28:19.019379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.019376 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerName="main" Oct 30 05:28:19.019588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.019542 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6ba582-05c4-4dd4-b7c5-98b0a2409bba" containerName="main" Oct 30 05:28:19.024686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.024666 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.041561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.041531 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-674082-openshift-default-dockercfg-g6ls2\"" Oct 30 05:28:19.056237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.047630 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:28:19.135873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.135793 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.135873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.135846 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.135954 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.135992 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2x2\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.136017 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.136103 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.136316 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.136337 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.136385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.136377 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.192161 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:19.192125 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:28:19.238389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238340 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238397 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238444 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238560 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238625 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238711 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238759 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2x2\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238763 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.238805 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238799 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.239076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238824 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.239076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238839 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.239076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.238914 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.239308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.239214 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.239308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.239254 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.240901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.240880 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.241032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.241013 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.260932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.260900 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.261146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.261121 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2x2\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-89swj\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.262288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.262250 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:19.262613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.262579 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:19.263743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.263724 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:28:19.359133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.357784 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:19.795180 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:28:19.795146 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1c7973_bf36_4bac_94de_1e67c84ad104.slice/crio-1b74859fb2221191c3a7d7d107eb49ee537e7b3f8f23216cb45f535472ca7338 WatchSource:0}: Error finding container 1b74859fb2221191c3a7d7d107eb49ee537e7b3f8f23216cb45f535472ca7338: Status 404 returned error can't find the container with id 1b74859fb2221191c3a7d7d107eb49ee537e7b3f8f23216cb45f535472ca7338 Oct 30 05:28:19.797413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.797390 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:28:19.797941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.797911 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:19.797990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.797976 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:19.798022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:19.798004 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:20.808243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:20.806330 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" event={"ID":"cd1c7973-bf36-4bac-94de-1e67c84ad104","Type":"ContainerStarted","Data":"fbd9a79e143e2b6571a2c92d48bb0cea614858a5cad563d5ddc33632c149b002"} Oct 30 05:28:20.808243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:20.806392 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" event={"ID":"cd1c7973-bf36-4bac-94de-1e67c84ad104","Type":"ContainerStarted","Data":"1b74859fb2221191c3a7d7d107eb49ee537e7b3f8f23216cb45f535472ca7338"} Oct 30 05:28:21.380251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:21.377300 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:21.388241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:21.381563 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:21.388241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:21.381632 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:22.361427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:22.361388 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:22.361836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:22.361455 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:23.361726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:23.361682 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:23.362122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:23.361745 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:24.199537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.199510 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:28:24.274259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274209 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.274533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274509 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:24.274656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274639 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.274743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274701 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.274807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274761 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274819 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274906 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.274971 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.275051 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.275083 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\") pod \"8de475be-4827-4da5-8563-16ca8beb390b\" (UID: \"8de475be-4827-4da5-8563-16ca8beb390b\") " Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.275457 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.275646 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.276191 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.276209 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.276245 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.277016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.276321 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:24.278571 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.277307 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:24.280491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.280449 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:24.283323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.283280 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podStartSLOduration=6.283265786 podStartE2EDuration="6.283265786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:20.879813168 +0000 UTC m=+2673.849308157" watchObservedRunningTime="2026-10-30 05:28:24.283265786 +0000 UTC m=+2677.252760773" Oct 30 05:28:24.284849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.284829 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "kube-api-access-9kkd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:24.287172 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.287150 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:28:24.288603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.288578 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy" pod "8de475be-4827-4da5-8563-16ca8beb390b" (UID: "8de475be-4827-4da5-8563-16ca8beb390b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:24.362077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.361973 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:24.363010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.362954 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:24.377517 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377484 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.377629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377520 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de475be-4827-4da5-8563-16ca8beb390b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.377629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377533 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.377629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377546 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kkd5\" (UniqueName: \"kubernetes.io/projected/8de475be-4827-4da5-8563-16ca8beb390b-kube-api-access-9kkd5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.377629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377558 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de475be-4827-4da5-8563-16ca8beb390b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.377629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.377566 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de475be-4827-4da5-8563-16ca8beb390b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:28:24.864101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.864065 5058 generic.go:356] "Generic (PLEG): container finished" podID="8de475be-4827-4da5-8563-16ca8beb390b" containerID="a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759" exitCode=0 Oct 30 05:28:24.864349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.864114 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" Oct 30 05:28:24.864349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.864142 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" event={"ID":"8de475be-4827-4da5-8563-16ca8beb390b","Type":"ContainerDied","Data":"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759"} Oct 30 05:28:24.864349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.864204 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr" event={"ID":"8de475be-4827-4da5-8563-16ca8beb390b","Type":"ContainerDied","Data":"50a2842108bf9c8be312499fb84ccbe033a4eb20754ee68b056242f0a86ff42d"} Oct 30 05:28:24.864349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.864247 5058 scope.go:122] "RemoveContainer" containerID="a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759" Oct 30 05:28:24.880396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.880370 5058 scope.go:122] "RemoveContainer" containerID="a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759" Oct 30 05:28:24.880829 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:24.880805 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759\": container with ID starting with a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759 not found: ID does not exist" containerID="a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759" Oct 30 05:28:24.880919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.880835 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759"} err="failed to get container status \"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759\": rpc error: code = NotFound desc = could not find container \"a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759\": container with ID starting with a2c6092570ea3b4b71edb91bde5a05bd6e3f51a3ae1f92b8196e40905dd38759 not found: ID does not exist" Oct 30 05:28:24.881999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.881973 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:28:24.883032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.883015 5058 cpu_manager.go:414] "containerMap: removing container" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" Oct 30 05:28:24.883100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.883033 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" Oct 30 05:28:24.883183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.883174 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de475be-4827-4da5-8563-16ca8beb390b" containerName="istio-proxy" Oct 30 05:28:24.888403 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.888386 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:24.906465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.906439 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ec92c7-serving-cert\"" Oct 30 05:28:24.907152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.907130 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:28:24.961885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.961774 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:28:24.968390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.968359 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr"] Oct 30 05:28:24.986207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.986174 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:24.986634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.986607 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:24.986829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.986811 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhv85\" (UniqueName: \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:24.986977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:24.986961 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.088759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.088716 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.088960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.088794 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhv85\" (UniqueName: \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.088960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.088830 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.088960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.088945 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.089176 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:25.089134 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-serving-cert: secret "maas-api-e2e-ipp-b-ec92c7-serving-cert" not found Oct 30 05:28:25.089295 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:25.089242 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls podName:f38f217e-7596-4095-b5ff-4e3d0ffa75fc nodeName:}" failed. No retries permitted until 2026-10-30 05:28:25.589209298 +0000 UTC m=+2678.558704268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls") pod "maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" (UID: "f38f217e-7596-4095-b5ff-4e3d0ffa75fc") : secret "maas-api-e2e-ipp-b-ec92c7-serving-cert" not found Oct 30 05:28:25.089491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.089473 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.091567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.091546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.112731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.112707 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhv85\" (UniqueName: \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.250243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.241784 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de475be-4827-4da5-8563-16ca8beb390b" path="/var/lib/kubelet/pods/8de475be-4827-4da5-8563-16ca8beb390b/volumes" Oct 30 05:28:25.362068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.362029 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:25.362282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.362094 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:25.617936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.617847 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.620333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.620311 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:25.804843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:25.804809 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:26.193507 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:28:26.193471 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38f217e_7596_4095_b5ff_4e3d0ffa75fc.slice/crio-b9bbe9d0b3b9b93a53be3d358c8d1c7b1a22fcf553d7442cdf9b86366f6cea98 WatchSource:0}: Error finding container b9bbe9d0b3b9b93a53be3d358c8d1c7b1a22fcf553d7442cdf9b86366f6cea98: Status 404 returned error can't find the container with id b9bbe9d0b3b9b93a53be3d358c8d1c7b1a22fcf553d7442cdf9b86366f6cea98 Oct 30 05:28:26.195420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.195401 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:28:26.198038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.198008 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:28:26.361677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.361584 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 05:28:26.361677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.361636 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 05:28:26.885575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.885475 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" event={"ID":"f38f217e-7596-4095-b5ff-4e3d0ffa75fc","Type":"ContainerStarted","Data":"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874"} Oct 30 05:28:26.885575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.885554 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" event={"ID":"f38f217e-7596-4095-b5ff-4e3d0ffa75fc","Type":"ContainerStarted","Data":"b9bbe9d0b3b9b93a53be3d358c8d1c7b1a22fcf553d7442cdf9b86366f6cea98"} Oct 30 05:28:26.886052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.885603 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:26.938048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:26.937991 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" podStartSLOduration=2.503164783 podStartE2EDuration="2.937976062s" totalImagesPullingTime="434.811279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:26.936238253 +0000 UTC m=+2679.905733239" watchObservedRunningTime="2026-10-30 05:28:26.937976062 +0000 UTC m=+2679.907471049" Oct 30 05:28:27.385239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:27.384933 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:27.387738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:27.386398 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:27.389451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:27.389417 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:28:31.509298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.509252 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:28:31.527054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.527027 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.534013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.533979 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:28:31.549618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.549586 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9dc17a-serving-cert\"" Oct 30 05:28:31.552541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.552513 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.552726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.552705 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.552821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.552803 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.552991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.552970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tl6t\" (UniqueName: \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.654575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.654536 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tl6t\" (UniqueName: \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.654751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.654607 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.654751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.654736 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.654902 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:31.654763 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-serving-cert: secret "maas-api-e2e-shared-b-w1-9dc17a-serving-cert" not found Oct 30 05:28:31.654902 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:31.654827 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls podName:33b9638d-bef8-40f5-88b7-3eca979ad2a5 nodeName:}" failed. No retries permitted until 2026-10-30 05:28:32.154807505 +0000 UTC m=+2685.124302491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls") pod "maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" (UID: "33b9638d-bef8-40f5-88b7-3eca979ad2a5") : secret "maas-api-e2e-shared-b-w1-9dc17a-serving-cert" not found Oct 30 05:28:31.654902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.654846 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.655563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.655538 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.657316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.657296 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:31.677952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:31.677925 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tl6t\" (UniqueName: \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:32.167704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:32.167664 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:32.170279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:32.170254 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:32.445626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:32.445539 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:32.852837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:32.852804 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:28:32.960445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:32.960408 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:28:33.003409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:33.003348 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" event={"ID":"33b9638d-bef8-40f5-88b7-3eca979ad2a5","Type":"ContainerStarted","Data":"45fa39a588cc305cc129e2a85d803c39714c2c38a5c3281c54c0383a340f0dae"} Oct 30 05:28:34.014747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:34.014713 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" event={"ID":"33b9638d-bef8-40f5-88b7-3eca979ad2a5","Type":"ContainerStarted","Data":"57a13ebd5b6d3027bce4819129e74e8fbcdb611b38319fcc5be2279597423d5b"} Oct 30 05:28:34.015174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:34.014790 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:34.062901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:34.062835 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" podStartSLOduration=2.718005051 podStartE2EDuration="3.062817768s" totalImagesPullingTime="344.812717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:34.061642733 +0000 UTC m=+2687.031137722" watchObservedRunningTime="2026-10-30 05:28:34.062817768 +0000 UTC m=+2687.032312759" Oct 30 05:28:40.031697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:40.031668 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:28:47.707363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:47.707335 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:28:47.721910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:47.721884 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:28:49.192991 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:28:49.192951 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:28:57.925117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:57.925084 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:28:57.931740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:57.931715 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:57.949477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:57.949450 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:28:57.958173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:57.958131 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2d21ab-serving-cert\"" Oct 30 05:28:58.070648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.070608 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.070820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.070658 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.070820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.070794 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.071179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.070971 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2xw\" (UniqueName: \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.173182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.173148 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.173393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.173207 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.173393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.173283 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.173531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.173427 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2xw\" (UniqueName: \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.174001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.173969 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.175863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.175810 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.175951 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.175897 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.195139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.195112 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2xw\" (UniqueName: \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\") pod \"maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.262516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.261257 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:28:58.632573 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:28:58.632543 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a6c749_7d46_413a_a440_6d6f579f0a7e.slice/crio-83b7415f61117a69ac2be88d7fea7f69c8dc7b54e4f563cfe9e44bc923931fb2 WatchSource:0}: Error finding container 83b7415f61117a69ac2be88d7fea7f69c8dc7b54e4f563cfe9e44bc923931fb2: Status 404 returned error can't find the container with id 83b7415f61117a69ac2be88d7fea7f69c8dc7b54e4f563cfe9e44bc923931fb2 Oct 30 05:28:58.634236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:58.634194 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:28:59.326293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:59.325196 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" event={"ID":"f5a6c749-7d46-413a-a440-6d6f579f0a7e","Type":"ContainerStarted","Data":"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4"} Oct 30 05:28:59.326293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:59.325281 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" event={"ID":"f5a6c749-7d46-413a-a440-6d6f579f0a7e","Type":"ContainerStarted","Data":"83b7415f61117a69ac2be88d7fea7f69c8dc7b54e4f563cfe9e44bc923931fb2"} Oct 30 05:28:59.326783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:28:59.326439 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:29:06.404262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:06.397621 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:29:06.453260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:06.451789 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" podStartSLOduration=9.042752655 podStartE2EDuration="9.451769553s" totalImagesPullingTime="409.016898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:59.384744548 +0000 UTC m=+2712.354239550" watchObservedRunningTime="2026-10-30 05:29:06.451769553 +0000 UTC m=+2719.421264542" Oct 30 05:29:15.640724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.640639 5058 generic.go:356] "Generic (PLEG): container finished" podID="914d0f0a-6044-4318-ad23-338d7e553377" containerID="068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c" exitCode=0 Oct 30 05:29:15.641260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.640714 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerDied","Data":"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c"} Oct 30 05:29:15.641330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.641313 5058 scope.go:122] "RemoveContainer" containerID="068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c" Oct 30 05:29:15.643293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.643257 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:15.643400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.643343 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:15.643442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:15.643396 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:16.651074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:16.651037 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerStarted","Data":"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7"} Oct 30 05:29:17.402325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:17.402293 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:29:17.403285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:17.403259 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 05:29:17.403405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:17.403334 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 05:29:17.807965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:17.807885 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:29:18.402944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:18.402900 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 05:29:18.403124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:18.402974 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 05:29:19.213525 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:29:19.213471 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:29:19.403417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:19.403379 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 05:29:19.403709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:19.403659 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 05:29:20.402444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:20.402409 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 05:29:20.402906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:20.402471 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 05:29:21.403096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:21.403057 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 05:29:21.403582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:21.403114 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 05:29:22.739103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:22.739047 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:29:22.740732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:22.740705 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:29:35.477373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:35.477342 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:29:49.242976 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:29:49.239697 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:29:50.499363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.499315 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" podUID="b7181c02-1b18-406a-bae2-a8323b5997c4" containerName="main" containerID="cri-o://7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2" gracePeriod=45 Oct 30 05:29:50.508982 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:29:50.508942 5058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7181c02_1b18_406a_bae2_a8323b5997c4.slice/crio-7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:29:50.790418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.790393 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:29:50.864285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864244 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.864484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864369 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.864568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864534 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\" (UniqueName: \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.864631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864567 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.864631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864577 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:50.864801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864777 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.864894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864872 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:50.864972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.864955 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\") pod \"b7181c02-1b18-406a-bae2-a8323b5997c4\" (UID: \"b7181c02-1b18-406a-bae2-a8323b5997c4\") " Oct 30 05:29:50.865316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.865141 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:50.865697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.865657 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:50.865788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.865704 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:50.865788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.865717 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:50.867414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.867385 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "kube-api-access-pg2bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:50.867509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.867415 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:50.867572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.867511 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm" pod "b7181c02-1b18-406a-bae2-a8323b5997c4" (UID: "b7181c02-1b18-406a-bae2-a8323b5997c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:50.967743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.967711 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2bc\" (UniqueName: \"kubernetes.io/projected/b7181c02-1b18-406a-bae2-a8323b5997c4-kube-api-access-pg2bc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:50.967743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.967741 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7181c02-1b18-406a-bae2-a8323b5997c4-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:50.967743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:50.967750 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7181c02-1b18-406a-bae2-a8323b5997c4-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:29:51.051064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.050975 5058 generic.go:356] "Generic (PLEG): container finished" podID="b7181c02-1b18-406a-bae2-a8323b5997c4" containerID="7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2" exitCode=0 Oct 30 05:29:51.051064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.051019 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" Oct 30 05:29:51.051291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.051059 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" event={"ID":"b7181c02-1b18-406a-bae2-a8323b5997c4","Type":"ContainerDied","Data":"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2"} Oct 30 05:29:51.051291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.051105 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k" event={"ID":"b7181c02-1b18-406a-bae2-a8323b5997c4","Type":"ContainerDied","Data":"3c23cd7e84edece034187ba841b391bc09151276e32c00f591f32647c0757e45"} Oct 30 05:29:51.051291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.051123 5058 scope.go:122] "RemoveContainer" containerID="7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2" Oct 30 05:29:51.063204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.063187 5058 scope.go:122] "RemoveContainer" containerID="7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2" Oct 30 05:29:51.063497 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:29:51.063479 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2\": container with ID starting with 7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2 not found: ID does not exist" containerID="7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2" Oct 30 05:29:51.063563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.063503 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2"} err="failed to get container status \"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2\": rpc error: code = NotFound desc = could not find container \"7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2\": container with ID starting with 7838356021f805ded7f8daaceffc381aa079d2e637846239e41fe9639ad249a2 not found: ID does not exist" Oct 30 05:29:51.115601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.115570 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:29:51.126901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.126869 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-44bc69/rate-test-model-44bc69-kserve-c59f6549-6mb2k"] Oct 30 05:29:51.163710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:29:51.163683 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7181c02-1b18-406a-bae2-a8323b5997c4" path="/var/lib/kubelet/pods/b7181c02-1b18-406a-bae2-a8323b5997c4/volumes" Oct 30 05:30:13.433155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.431295 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:30:13.435036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.434435 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b7181c02-1b18-406a-bae2-a8323b5997c4" containerName="main" Oct 30 05:30:13.435036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.434474 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7181c02-1b18-406a-bae2-a8323b5997c4" containerName="main" Oct 30 05:30:13.435036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.434722 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7181c02-1b18-406a-bae2-a8323b5997c4" containerName="main" Oct 30 05:30:13.442054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.442034 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:30:13.442256 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.442150 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.468828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.468754 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec92c7-dockercfg-w9d9z\"" Oct 30 05:30:13.482436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.482402 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec92c7\"" Oct 30 05:30:13.531556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.531519 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.532097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.532063 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjqn\" (UniqueName: \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.634163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.634123 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjqn\" (UniqueName: \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.634385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.634327 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.635324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.635298 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.657164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.657132 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjqn\" (UniqueName: \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:13.758959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:13.758878 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:14.069968 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:30:14.069939 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286f49e3_ea84_473f_84d1_67c9237a56e3.slice/crio-14984ab514798d7f1d34b259c6777eee318e3d043ef63a5c5c4c7ba41b98908a WatchSource:0}: Error finding container 14984ab514798d7f1d34b259c6777eee318e3d043ef63a5c5c4c7ba41b98908a: Status 404 returned error can't find the container with id 14984ab514798d7f1d34b259c6777eee318e3d043ef63a5c5c4c7ba41b98908a Oct 30 05:30:14.070793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:14.070768 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:30:14.532921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:14.532884 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" event={"ID":"286f49e3-ea84-473f-84d1-67c9237a56e3","Type":"ContainerStarted","Data":"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2"} Oct 30 05:30:14.533342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:14.532941 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" event={"ID":"286f49e3-ea84-473f-84d1-67c9237a56e3","Type":"ContainerStarted","Data":"14984ab514798d7f1d34b259c6777eee318e3d043ef63a5c5c4c7ba41b98908a"} Oct 30 05:30:14.533342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:14.532970 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:14.578429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:14.578377 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" podStartSLOduration=1.57836351 podStartE2EDuration="1.57836351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:14.575772569 +0000 UTC m=+2787.545267556" watchObservedRunningTime="2026-10-30 05:30:14.57836351 +0000 UTC m=+2787.547858498" Oct 30 05:30:19.192295 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:30:19.192256 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:30:25.542704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:25.542671 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:30:35.934359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.934325 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:30:35.941041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.941014 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:35.957944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.957913 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:30:35.964809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.964786 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"kube-root-ca.crt\"" Oct 30 05:30:35.964966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.964903 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"openshift-service-ca.crt\"" Oct 30 05:30:35.965232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.965201 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"default-dockercfg-rrhv2\"" Oct 30 05:30:35.977328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:35.977307 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"rate-test-model-674082-kserve-self-signed-certs\"" Oct 30 05:30:36.092310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092280 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.092474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092390 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.092474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092429 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.092474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092453 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.092585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092515 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzbq\" (UniqueName: \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.092585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.092559 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.197872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.197775 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.206023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.198181 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.206322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.206296 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.216820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.216788 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.217042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.217021 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzbq\" (UniqueName: \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.217238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.217193 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.224255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.217983 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.224847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.200728 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.225010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.206898 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.275375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.274808 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.379239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.372376 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzbq\" (UniqueName: \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.379239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.372745 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\") pod \"rate-test-model-674082-kserve-fb558df85-gc4pk\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.560377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.560298 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:36.878681 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:30:36.878653 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1314cab_6d61_40cf_98f7_be85d74b1f39.slice/crio-4dc80b57d3fb1afb0a1402d4c2a5136dbd2c7969826a8dbdd79685c77f71fadb WatchSource:0}: Error finding container 4dc80b57d3fb1afb0a1402d4c2a5136dbd2c7969826a8dbdd79685c77f71fadb: Status 404 returned error can't find the container with id 4dc80b57d3fb1afb0a1402d4c2a5136dbd2c7969826a8dbdd79685c77f71fadb Oct 30 05:30:36.881013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:36.880988 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:30:37.800259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:37.800184 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" event={"ID":"b1314cab-6d61-40cf-98f7-be85d74b1f39","Type":"ContainerStarted","Data":"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7"} Oct 30 05:30:37.800259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:37.800245 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" event={"ID":"b1314cab-6d61-40cf-98f7-be85d74b1f39","Type":"ContainerStarted","Data":"4dc80b57d3fb1afb0a1402d4c2a5136dbd2c7969826a8dbdd79685c77f71fadb"} Oct 30 05:30:37.850035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:37.849983 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" podStartSLOduration=2.849969757 podStartE2EDuration="2.849969757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:37.846179942 +0000 UTC m=+2810.815674929" watchObservedRunningTime="2026-10-30 05:30:37.849969757 +0000 UTC m=+2810.819464744" Oct 30 05:30:38.560525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:38.560488 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:46.560455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:46.560421 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:46.577549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:46.577521 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:46.897767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:46.897740 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:30:49.191255 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:30:49.191191 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:30:51.870346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.870312 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:30:51.877913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.877886 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:51.896177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.896148 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:30:51.904162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.904138 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9dc17a\"/\"kube-root-ca.crt\"" Oct 30 05:30:51.904347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.904170 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9dc17a\"/\"openshift-service-ca.crt\"" Oct 30 05:30:51.904347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.904250 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9dc17a\"/\"auth-test-model-9dc17a-kserve-self-signed-certs\"" Oct 30 05:30:51.909325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:51.909301 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9dc17a\"/\"default-dockercfg-sh59n\"" Oct 30 05:30:52.025931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.025899 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.026112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.025969 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw8b\" (UniqueName: \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.026112 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.026068 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.026202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.026165 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.026351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.026327 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.026405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.026372 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.128168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.128092 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.157792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.157739 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.158782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.158760 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.159626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.159605 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.160457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.160438 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.160703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.160684 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw8b\" (UniqueName: \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.161482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.159183 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.185638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.182876 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.185860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.183298 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.185977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.158602 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.192487 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.192457 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.197921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.197894 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw8b\" (UniqueName: \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\") pod \"auth-test-model-9dc17a-kserve-6c877997c7-rxdpx\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.220108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.220077 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:52.663192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.663160 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:30:52.664098 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:30:52.664069 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58ef2fa7_5a82_4019_89b3_39595004d62c.slice/crio-526a44162a2a294214fe038b21500dee1ae30a3ce02de58e0a4139f204e03cae WatchSource:0}: Error finding container 526a44162a2a294214fe038b21500dee1ae30a3ce02de58e0a4139f204e03cae: Status 404 returned error can't find the container with id 526a44162a2a294214fe038b21500dee1ae30a3ce02de58e0a4139f204e03cae Oct 30 05:30:52.939182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.939104 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" event={"ID":"58ef2fa7-5a82-4019-89b3-39595004d62c","Type":"ContainerStarted","Data":"3423212e92be73794f14770a8d90b916f9df19aa0acb359959b9b72e303fafd3"} Oct 30 05:30:52.939182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.939144 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" event={"ID":"58ef2fa7-5a82-4019-89b3-39595004d62c","Type":"ContainerStarted","Data":"526a44162a2a294214fe038b21500dee1ae30a3ce02de58e0a4139f204e03cae"} Oct 30 05:30:52.990495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:52.990438 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" podStartSLOduration=1.990421751 podStartE2EDuration="1.990421751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:52.987063715 +0000 UTC m=+2825.956558703" watchObservedRunningTime="2026-10-30 05:30:52.990421751 +0000 UTC m=+2825.959916740" Oct 30 05:30:53.221750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:53.221677 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:30:59.269304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.269211 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-8ac5bc"] Oct 30 05:30:59.274590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.274566 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:30:59.289330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.289306 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-8ac5bc"] Oct 30 05:30:59.296570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.296548 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\"" Oct 30 05:30:59.401849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.401808 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtbp\" (UniqueName: \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\") pod \"test-curl-8876-8ac5bc\" (UID: \"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4\") " pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:30:59.503207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.503173 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtbp\" (UniqueName: \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\") pod \"test-curl-8876-8ac5bc\" (UID: \"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4\") " pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:30:59.525027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.524960 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtbp\" (UniqueName: \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\") pod \"test-curl-8876-8ac5bc\" (UID: \"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4\") " pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:30:59.590540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.590505 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:30:59.890116 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:30:59.890081 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9e2d30d_dd78_4ab5_b12e_857e4711b9c4.slice/crio-9e2fb109860f3e5bc903d87fdc292b08ae4aa0bd0dff54277d8c1d5611966aa2 WatchSource:0}: Error finding container 9e2fb109860f3e5bc903d87fdc292b08ae4aa0bd0dff54277d8c1d5611966aa2: Status 404 returned error can't find the container with id 9e2fb109860f3e5bc903d87fdc292b08ae4aa0bd0dff54277d8c1d5611966aa2 Oct 30 05:30:59.892397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:30:59.892368 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-8ac5bc"] Oct 30 05:31:00.045631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:00.045576 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-8ac5bc" event={"ID":"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4","Type":"ContainerStarted","Data":"bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8"} Oct 30 05:31:00.045631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:00.045629 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-8ac5bc" event={"ID":"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4","Type":"ContainerStarted","Data":"9e2fb109860f3e5bc903d87fdc292b08ae4aa0bd0dff54277d8c1d5611966aa2"} Oct 30 05:31:00.089836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:00.089788 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-8ac5bc" podStartSLOduration=1.089773876 podStartE2EDuration="1.089773876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:00.08776685 +0000 UTC m=+2833.057261838" watchObservedRunningTime="2026-10-30 05:31:00.089773876 +0000 UTC m=+2833.059268863" Oct 30 05:31:00.958359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:00.958323 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-8ac5bc"] Oct 30 05:31:00.963262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:00.963211 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-8ac5bc"] Oct 30 05:31:01.135176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.135145 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-3723ff"] Oct 30 05:31:01.136058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.136041 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" containerName="test-curl-8876-8ac5bc" Oct 30 05:31:01.136114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.136059 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" containerName="test-curl-8876-8ac5bc" Oct 30 05:31:01.136207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.136197 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" containerName="test-curl-8876-8ac5bc" Oct 30 05:31:01.140210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.140191 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:01.168165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.168137 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-3723ff"] Oct 30 05:31:01.233926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.233833 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhtw\" (UniqueName: \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\") pod \"test-curl-8876-3723ff\" (UID: \"d4aec061-e4d2-476c-a2cd-02bb6b1a0224\") " pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:01.335330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.335298 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhtw\" (UniqueName: \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\") pod \"test-curl-8876-3723ff\" (UID: \"d4aec061-e4d2-476c-a2cd-02bb6b1a0224\") " pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:01.356873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.356846 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhtw\" (UniqueName: \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\") pod \"test-curl-8876-3723ff\" (UID: \"d4aec061-e4d2-476c-a2cd-02bb6b1a0224\") " pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:01.454651 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.454619 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:01.764966 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:31:01.764903 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4aec061_e4d2_476c_a2cd_02bb6b1a0224.slice/crio-692e26a125bf8209425a2db0406ccc91a2708930851cf15a4653390a6224d704 WatchSource:0}: Error finding container 692e26a125bf8209425a2db0406ccc91a2708930851cf15a4653390a6224d704: Status 404 returned error can't find the container with id 692e26a125bf8209425a2db0406ccc91a2708930851cf15a4653390a6224d704 Oct 30 05:31:01.767444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:01.767417 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-3723ff"] Oct 30 05:31:02.061626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.061546 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-3723ff" event={"ID":"d4aec061-e4d2-476c-a2cd-02bb6b1a0224","Type":"ContainerStarted","Data":"72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4"} Oct 30 05:31:02.061626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.061595 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-3723ff" event={"ID":"d4aec061-e4d2-476c-a2cd-02bb6b1a0224","Type":"ContainerStarted","Data":"692e26a125bf8209425a2db0406ccc91a2708930851cf15a4653390a6224d704"} Oct 30 05:31:02.062056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.061626 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-8ac5bc" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" containerName="test-curl-8876-8ac5bc" containerID="cri-o://bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8" gracePeriod=2 Oct 30 05:31:02.106427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.106381 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-3723ff" podStartSLOduration=1.106352697 podStartE2EDuration="1.106352697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:02.101651205 +0000 UTC m=+2835.071146174" watchObservedRunningTime="2026-10-30 05:31:02.106352697 +0000 UTC m=+2835.075847686" Oct 30 05:31:02.221207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.221180 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:31:02.281955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:02.281922 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:31:03.016474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:03.016418 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-3723ff"] Oct 30 05:31:03.020907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:03.020878 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-3723ff"] Oct 30 05:31:03.099686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:03.099636 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:31:04.090834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:04.090803 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-3723ff" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" containerName="test-curl-8876-3723ff" containerID="cri-o://72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4" gracePeriod=2 Oct 30 05:31:04.289196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:04.289168 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:31:04.393444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:04.393421 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\" (UniqueName: \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\") pod \"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4\" (UID: \"c9e2d30d-dd78-4ab5-b12e-857e4711b9c4\") " Oct 30 05:31:04.395794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:04.395769 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp" pod "c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" (UID: "c9e2d30d-dd78-4ab5-b12e-857e4711b9c4"). InnerVolumeSpecName "kube-api-access-fwtbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:04.496114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:04.496074 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtbp\" (UniqueName: \"kubernetes.io/projected/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4-kube-api-access-fwtbp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:05.101067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.101027 5058 generic.go:356] "Generic (PLEG): container finished" podID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" containerID="bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8" exitCode=137 Oct 30 05:31:05.101301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.101088 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:31:05.101301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.101096 5058 scope.go:122] "RemoveContainer" containerID="bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8" Oct 30 05:31:05.112551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.112533 5058 scope.go:122] "RemoveContainer" containerID="bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8" Oct 30 05:31:05.112794 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:05.112777 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8\": container with ID starting with bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8 not found: ID does not exist" containerID="bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8" Oct 30 05:31:05.112843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.112801 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8"} err="failed to get container status \"bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8\": rpc error: code = NotFound desc = could not find container \"bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8\": container with ID starting with bbd54e42f13bd28ee2a507711eb7b80103b40f3b7a3fabcd66d19f178d77eda8 not found: ID does not exist" Oct 30 05:31:05.125787 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:05.125751 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-8ac5bc\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" pod="opendatahub/test-curl-8876-8ac5bc" Oct 30 05:31:05.164884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:05.164850 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e2d30d-dd78-4ab5-b12e-857e4711b9c4" path="/var/lib/kubelet/pods/c9e2d30d-dd78-4ab5-b12e-857e4711b9c4/volumes" Oct 30 05:31:06.321628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:06.321602 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:06.328778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:06.328756 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\" (UniqueName: \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\") pod \"d4aec061-e4d2-476c-a2cd-02bb6b1a0224\" (UID: \"d4aec061-e4d2-476c-a2cd-02bb6b1a0224\") " Oct 30 05:31:06.330753 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:06.330732 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw" pod "d4aec061-e4d2-476c-a2cd-02bb6b1a0224" (UID: "d4aec061-e4d2-476c-a2cd-02bb6b1a0224"). InnerVolumeSpecName "kube-api-access-6hhtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:06.430993 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:06.430965 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hhtw\" (UniqueName: \"kubernetes.io/projected/d4aec061-e4d2-476c-a2cd-02bb6b1a0224-kube-api-access-6hhtw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:07.120961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.120926 5058 generic.go:356] "Generic (PLEG): container finished" podID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" containerID="72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4" exitCode=137 Oct 30 05:31:07.121176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.120971 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:07.121176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.121048 5058 scope.go:122] "RemoveContainer" containerID="72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4" Oct 30 05:31:07.132854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.132669 5058 scope.go:122] "RemoveContainer" containerID="72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4" Oct 30 05:31:07.132931 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:07.132913 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4\": container with ID starting with 72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4 not found: ID does not exist" containerID="72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4" Oct 30 05:31:07.132987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.132940 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4"} err="failed to get container status \"72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4\": rpc error: code = NotFound desc = could not find container \"72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4\": container with ID starting with 72e250f9490e5aa7f37ae70057fd345073d9d01f85b5bb3cc37daa17de6dd6e4 not found: ID does not exist" Oct 30 05:31:07.144364 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:07.144335 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-3723ff\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:07.163579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:07.163547 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" path="/var/lib/kubelet/pods/d4aec061-e4d2-476c-a2cd-02bb6b1a0224/volumes" Oct 30 05:31:07.173167 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:07.173145 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-3723ff\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" pod="opendatahub/test-curl-8876-3723ff" Oct 30 05:31:19.191630 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:19.191592 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:31:30.432815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.432781 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:31:30.433368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.433008 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" podUID="26397e20-d69e-4c5e-bc48-861c36272a84" containerName="maas-api" containerID="cri-o://b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2" gracePeriod=30 Oct 30 05:31:30.649142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.649108 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:31:30.649398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.649373 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" podUID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" containerName="payload-pre-processing" containerID="cri-o://a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98" gracePeriod=30 Oct 30 05:31:30.797849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.794069 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:31:30.828744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.828711 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\") pod \"26397e20-d69e-4c5e-bc48-861c36272a84\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " Oct 30 05:31:30.828936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.828780 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\" (UniqueName: \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\") pod \"26397e20-d69e-4c5e-bc48-861c36272a84\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " Oct 30 05:31:30.828936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.828915 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\") pod \"26397e20-d69e-4c5e-bc48-861c36272a84\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " Oct 30 05:31:30.829070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.828998 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\") pod \"26397e20-d69e-4c5e-bc48-861c36272a84\" (UID: \"26397e20-d69e-4c5e-bc48-861c36272a84\") " Oct 30 05:31:30.830128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.830099 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca" pod "26397e20-d69e-4c5e-bc48-861c36272a84" (UID: "26397e20-d69e-4c5e-bc48-861c36272a84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:30.833337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.833309 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls" pod "26397e20-d69e-4c5e-bc48-861c36272a84" (UID: "26397e20-d69e-4c5e-bc48-861c36272a84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:30.834594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.834569 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs" pod "26397e20-d69e-4c5e-bc48-861c36272a84" (UID: "26397e20-d69e-4c5e-bc48-861c36272a84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:30.834709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.834592 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq" pod "26397e20-d69e-4c5e-bc48-861c36272a84" (UID: "26397e20-d69e-4c5e-bc48-861c36272a84"). InnerVolumeSpecName "kube-api-access-7gnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:30.857831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.857799 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:31:30.859173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859148 5058 cpu_manager.go:414] "containerMap: removing container" podUID="26397e20-d69e-4c5e-bc48-861c36272a84" containerName="maas-api" Oct 30 05:31:30.859283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859174 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26397e20-d69e-4c5e-bc48-861c36272a84" containerName="maas-api" Oct 30 05:31:30.859283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859246 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" containerName="test-curl-8876-3723ff" Oct 30 05:31:30.859283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859254 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" containerName="test-curl-8876-3723ff" Oct 30 05:31:30.859489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859475 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="26397e20-d69e-4c5e-bc48-861c36272a84" containerName="maas-api" Oct 30 05:31:30.859525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.859500 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4aec061-e4d2-476c-a2cd-02bb6b1a0224" containerName="test-curl-8876-3723ff" Oct 30 05:31:30.866490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.866466 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.883801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.883657 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2d21ab-openshift-default-dockercfg-85kql\"" Oct 30 05:31:30.887873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.887840 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931204 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931278 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931450 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931471 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqfp\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931547 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931566 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931603 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931651 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931788 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931800 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26397e20-d69e-4c5e-bc48-861c36272a84-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931815 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gnbq\" (UniqueName: \"kubernetes.io/projected/26397e20-d69e-4c5e-bc48-861c36272a84-kube-api-access-7gnbq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.931826 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26397e20-d69e-4c5e-bc48-861c36272a84-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.936698 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:31:30.939259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.936877 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" podUID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" containerName="maas-api" containerID="cri-o://fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29" gracePeriod=30 Oct 30 05:31:30.949101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:30.949077 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:31:31.033302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033184 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\" (UniqueName: \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\") pod \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033348 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\") pod \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\" (UID: \"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f\") " Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033735 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume" pod "a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" (UID: "a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033870 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033914 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.033956 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034004 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034068 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034146 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034172 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034440 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034474 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqfp\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.034757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034619 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.035674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.034627 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.035674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.035047 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.035924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.035899 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.037148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.037093 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.037355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.037315 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.037929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.037903 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.038941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.038917 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.040204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.040164 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8" pod "a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" (UID: "a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f"). InnerVolumeSpecName "kube-api-access-mbjr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:31.057189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.057162 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.057377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.057356 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqfp\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.151697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.151660 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjr8\" (UniqueName: \"kubernetes.io/projected/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f-kube-api-access-mbjr8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.178038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.178005 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:31:31.178281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.178250 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" podUID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" containerName="payload-pre-processing" containerID="cri-o://093e970481a98d6101013f05c80356ef3c62d4582b946e381ad6d4cacdac0937" gracePeriod=30 Oct 30 05:31:31.198628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.198597 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:31.312018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.311779 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:31:31.356582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.356158 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\" (UniqueName: \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\") pod \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " Oct 30 05:31:31.358443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.357205 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca" pod "a0d11cb1-651d-4dec-a4f4-0f7e98c34506" (UID: "a0d11cb1-651d-4dec-a4f4-0f7e98c34506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:31.358443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.357268 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\") pod \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " Oct 30 05:31:31.358443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.358094 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.363916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.363878 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt" pod "a0d11cb1-651d-4dec-a4f4-0f7e98c34506" (UID: "a0d11cb1-651d-4dec-a4f4-0f7e98c34506"). InnerVolumeSpecName "kube-api-access-gwdwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:31.388849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.388037 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.389857 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" containerName="maas-api" Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.389881 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" containerName="maas-api" Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.389899 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" containerName="payload-pre-processing" Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.389906 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" containerName="payload-pre-processing" Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.390148 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" containerName="maas-api" Oct 30 05:31:31.390508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.390160 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" containerName="payload-pre-processing" Oct 30 05:31:31.404633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.404608 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.405899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.405868 5058 generic.go:356] "Generic (PLEG): container finished" podID="26397e20-d69e-4c5e-bc48-861c36272a84" containerID="b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2" exitCode=0 Oct 30 05:31:31.406099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.406069 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" event={"ID":"26397e20-d69e-4c5e-bc48-861c36272a84","Type":"ContainerDied","Data":"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2"} Oct 30 05:31:31.406190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.406122 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" event={"ID":"26397e20-d69e-4c5e-bc48-861c36272a84","Type":"ContainerDied","Data":"e47bfee22156dc604f7e37ccdac805b84ab64d543b13c246e0e50595a89f6217"} Oct 30 05:31:31.406190 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.406153 5058 scope.go:122] "RemoveContainer" containerID="b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2" Oct 30 05:31:31.406481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.406341 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx" Oct 30 05:31:31.417373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.416343 5058 generic.go:356] "Generic (PLEG): container finished" podID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" containerID="fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29" exitCode=0 Oct 30 05:31:31.417373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.416517 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" event={"ID":"a0d11cb1-651d-4dec-a4f4-0f7e98c34506","Type":"ContainerDied","Data":"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29"} Oct 30 05:31:31.417373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.416563 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" event={"ID":"a0d11cb1-651d-4dec-a4f4-0f7e98c34506","Type":"ContainerDied","Data":"a6d719bef1a83c9e95ede9c96b6a547af0c58e2a66e49db1aa77d778dd651cb9"} Oct 30 05:31:31.417373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.416639 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9" Oct 30 05:31:31.420344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.419163 5058 generic.go:356] "Generic (PLEG): container finished" podID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" containerID="a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98" exitCode=0 Oct 30 05:31:31.420344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.419254 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" event={"ID":"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f","Type":"ContainerDied","Data":"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98"} Oct 30 05:31:31.420344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.419294 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" event={"ID":"a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f","Type":"ContainerDied","Data":"8c86efa5c30e12089b912dc4e114687cef87f92d16a1bea3c9d4ef70b0098410"} Oct 30 05:31:31.420344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.419381 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr" Oct 30 05:31:31.420344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.419973 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:31:31.423253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.422608 5058 generic.go:356] "Generic (PLEG): container finished" podID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" containerID="093e970481a98d6101013f05c80356ef3c62d4582b946e381ad6d4cacdac0937" exitCode=0 Oct 30 05:31:31.423253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.422671 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" event={"ID":"71b3028b-d4ca-4704-baa7-778e70a0ccdd","Type":"ContainerDied","Data":"093e970481a98d6101013f05c80356ef3c62d4582b946e381ad6d4cacdac0937"} Oct 30 05:31:31.425382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.425330 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-02cffa-serving-cert\"" Oct 30 05:31:31.430969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.430941 5058 scope.go:122] "RemoveContainer" containerID="b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2" Oct 30 05:31:31.431416 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:31.431384 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2\": container with ID starting with b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2 not found: ID does not exist" containerID="b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2" Oct 30 05:31:31.431515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.431425 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2"} err="failed to get container status \"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2\": rpc error: code = NotFound desc = could not find container \"b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2\": container with ID starting with b1ec370a3655d141e3a853a5fc3bf8a4d51d9c3a1d9721ea1fda422006f1fbc2 not found: ID does not exist" Oct 30 05:31:31.431515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.431447 5058 scope.go:122] "RemoveContainer" containerID="fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29" Oct 30 05:31:31.444484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.444462 5058 scope.go:122] "RemoveContainer" containerID="fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29" Oct 30 05:31:31.445156 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:31.445040 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29\": container with ID starting with fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29 not found: ID does not exist" containerID="fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29" Oct 30 05:31:31.445156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.445068 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29"} err="failed to get container status \"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29\": rpc error: code = NotFound desc = could not find container \"fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29\": container with ID starting with fc20e3959bf8192a0bb128f6334438adac33d097c8d9f3f40a8e483fd79a2b29 not found: ID does not exist" Oct 30 05:31:31.445156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.445085 5058 scope.go:122] "RemoveContainer" containerID="a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98" Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465018 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\") pod \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465152 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\") pod \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\" (UID: \"a0d11cb1-651d-4dec-a4f4-0f7e98c34506\") " Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465532 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465609 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465710 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rxv\" (UniqueName: \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.465944 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.466351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.466020 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdwt\" (UniqueName: \"kubernetes.io/projected/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-kube-api-access-gwdwt\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.468241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.467867 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls" pod "a0d11cb1-651d-4dec-a4f4-0f7e98c34506" (UID: "a0d11cb1-651d-4dec-a4f4-0f7e98c34506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:31.469877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.469839 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs" pod "a0d11cb1-651d-4dec-a4f4-0f7e98c34506" (UID: "a0d11cb1-651d-4dec-a4f4-0f7e98c34506"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:31.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.505766 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:31:31.514239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.505833 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3855-78d94d8b5b-vkppx"] Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.601690 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.601840 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.601931 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.602052 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rxv\" (UniqueName: \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.602170 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.602186 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d11cb1-651d-4dec-a4f4-0f7e98c34506-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.664792 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.688142 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.712242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.700549 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rxv\" (UniqueName: \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.727399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.727353 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:31.727536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.727468 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:31.727536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.727511 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:31.762853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.761815 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:31:31.775350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.775317 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:31.859919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.859696 5058 scope.go:122] "RemoveContainer" containerID="a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98" Oct 30 05:31:31.861426 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:31.860325 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98\": container with ID starting with a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98 not found: ID does not exist" containerID="a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98" Oct 30 05:31:31.861426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.860371 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98"} err="failed to get container status \"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98\": rpc error: code = NotFound desc = could not find container \"a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98\": container with ID starting with a1416ccaf584c4e7eee87af0700a63a680306dd7e1c6c38896a29dc4887e7b98 not found: ID does not exist" Oct 30 05:31:31.875148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.875104 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:31:31.887459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.887195 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr"] Oct 30 05:31:31.900096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.900064 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:31:31.904342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.904314 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-60a531-fd55479f7-47qs9"] Oct 30 05:31:31.913960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.913935 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:31:31.984388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.984351 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:31:31.985601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.985578 5058 cpu_manager.go:414] "containerMap: removing container" podUID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" containerName="payload-pre-processing" Oct 30 05:31:31.985601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.985601 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" containerName="payload-pre-processing" Oct 30 05:31:31.985888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.985873 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" containerName="payload-pre-processing" Oct 30 05:31:31.990434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:31.990412 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.019263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.019022 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\") pod \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " Oct 30 05:31:32.019666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.019642 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\" (UniqueName: \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\") pod \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\" (UID: \"71b3028b-d4ca-4704-baa7-778e70a0ccdd\") " Oct 30 05:31:32.020153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020131 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume" pod "71b3028b-d4ca-4704-baa7-778e70a0ccdd" (UID: "71b3028b-d4ca-4704-baa7-778e70a0ccdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:32.020285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020264 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020421 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsfw\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020455 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020524 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020646 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020696 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020763 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020822 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.020894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.020853 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.021063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.021048 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b3028b-d4ca-4704-baa7-778e70a0ccdd-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:32.024892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.024869 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg" pod "71b3028b-d4ca-4704-baa7-778e70a0ccdd" (UID: "71b3028b-d4ca-4704-baa7-778e70a0ccdd"). InnerVolumeSpecName "kube-api-access-5tzrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:32.026181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.026161 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:31:32.039770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.039747 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:32.123300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.122932 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123037 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsfw\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123066 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123131 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123300 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123200 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123360 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123430 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123489 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.123668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.123517 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.124685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.124660 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.125877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.125161 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.125877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.125452 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.125877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.125488 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tzrg\" (UniqueName: \"kubernetes.io/projected/71b3028b-d4ca-4704-baa7-778e70a0ccdd-kube-api-access-5tzrg\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:32.125877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.125744 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.126121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.125971 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.128715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.128685 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.129573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.129532 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.163297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.163262 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsfw\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.208890 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.208546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.363235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.360054 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:32.443953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.442797 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" event={"ID":"7b57f9b3-b940-46df-8e3d-10509d96fab0","Type":"ContainerStarted","Data":"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9"} Oct 30 05:31:32.443953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.442855 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" event={"ID":"7b57f9b3-b940-46df-8e3d-10509d96fab0","Type":"ContainerStarted","Data":"74b1feb4d55ed625f46a0f83aca11ae1bd2cae274c6fe10bda3337201a22e9a7"} Oct 30 05:31:32.443953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.443772 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:31:32.450809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.447331 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" event={"ID":"71b3028b-d4ca-4704-baa7-778e70a0ccdd","Type":"ContainerDied","Data":"6da6d8f119b82a82f1ef4b0866dd0813ceb83d6422793698b18837c25d9efb9c"} Oct 30 05:31:32.450809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.447382 5058 scope.go:122] "RemoveContainer" containerID="093e970481a98d6101013f05c80356ef3c62d4582b946e381ad6d4cacdac0937" Oct 30 05:31:32.450809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.447480 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4" Oct 30 05:31:32.602816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.602742 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podStartSLOduration=29.602720209 podStartE2EDuration="29.602720209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:32.548676116 +0000 UTC m=+2865.518171104" watchObservedRunningTime="2026-10-30 05:31:32.602720209 +0000 UTC m=+2865.572215195" Oct 30 05:31:32.620323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.620280 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:31:32.625315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.625202 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4"] Oct 30 05:31:32.760716 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:31:32.760682 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d0cb58_f059_4b15_83bd_6e37a907234c.slice/crio-7e55bc81502dd170409ef323ccb65d1d41c3796c8c8218ff039fe86b52216ec0 WatchSource:0}: Error finding container 7e55bc81502dd170409ef323ccb65d1d41c3796c8c8218ff039fe86b52216ec0: Status 404 returned error can't find the container with id 7e55bc81502dd170409ef323ccb65d1d41c3796c8c8218ff039fe86b52216ec0 Oct 30 05:31:32.761159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.761138 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:31:32.763394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.763366 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:32.763455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.763426 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:32.763508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:32.763460 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:33.170960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.170929 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26397e20-d69e-4c5e-bc48-861c36272a84" path="/var/lib/kubelet/pods/26397e20-d69e-4c5e-bc48-861c36272a84/volumes" Oct 30 05:31:33.171439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.171418 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b3028b-d4ca-4704-baa7-778e70a0ccdd" path="/var/lib/kubelet/pods/71b3028b-d4ca-4704-baa7-778e70a0ccdd/volumes" Oct 30 05:31:33.172073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.172049 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d11cb1-651d-4dec-a4f4-0f7e98c34506" path="/var/lib/kubelet/pods/a0d11cb1-651d-4dec-a4f4-0f7e98c34506/volumes" Oct 30 05:31:33.172916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.172893 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f" path="/var/lib/kubelet/pods/a4aa4bd2-7c5f-4ef4-b171-ff20b1c8b79f/volumes" Oct 30 05:31:33.202292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.202260 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:33.202679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.202652 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:31:33.205353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.205319 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 05:31:33.472520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.472473 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" event={"ID":"41d0cb58-f059-4b15-83bd-6e37a907234c","Type":"ContainerStarted","Data":"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898"} Oct 30 05:31:33.473193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.473165 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" event={"ID":"41d0cb58-f059-4b15-83bd-6e37a907234c","Type":"ContainerStarted","Data":"7e55bc81502dd170409ef323ccb65d1d41c3796c8c8218ff039fe86b52216ec0"} Oct 30 05:31:33.477495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.477467 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" event={"ID":"bbe7d883-0a03-41ad-ad68-1698f4a650b6","Type":"ContainerStarted","Data":"332cb9f2dc7ee0c91a7853c2c9ab17f7002ca0a9e0398cd838ec8fb77f8c49da"} Oct 30 05:31:33.477586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.477518 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" event={"ID":"bbe7d883-0a03-41ad-ad68-1698f4a650b6","Type":"ContainerStarted","Data":"1cdb5390f035dc8c31a04e82a36c4baca97f51bc34684aba0b2d32d438cd9e7c"} Oct 30 05:31:33.560691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.560630 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podStartSLOduration=61.560614342 podStartE2EDuration="1m1.560614342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:33.525809281 +0000 UTC m=+2866.495304272" watchObservedRunningTime="2026-10-30 05:31:33.560614342 +0000 UTC m=+2866.530109329" Oct 30 05:31:33.577903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:33.577842 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" podStartSLOduration=40.935893203 podStartE2EDuration="41.577826299s" totalImagesPullingTime="641.933096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:33.575828144 +0000 UTC m=+2866.545323135" watchObservedRunningTime="2026-10-30 05:31:33.577826299 +0000 UTC m=+2866.547321288" Oct 30 05:31:34.199653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.199616 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:31:34.199835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.199682 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 05:31:34.363888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.363852 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:34.368392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.368359 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 05:31:34.368624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.368601 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 05:31:34.485986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:34.485913 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:35.199720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:35.199687 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:31:35.199904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:35.199735 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 05:31:35.366243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:35.365927 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 05:31:35.366243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:35.365986 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 05:31:36.200007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:36.199965 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:31:36.200462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:36.200030 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 05:31:36.374243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:36.374058 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 05:31:36.374243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:36.374119 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 05:31:37.200159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:37.200126 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:31:37.200612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:37.200176 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 05:31:37.374284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:37.365296 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 05:31:37.374284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:37.365356 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 05:31:38.220861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:38.220825 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:38.221519 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:38.221489 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:38.222409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:38.222385 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:31:38.362176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:38.362144 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 05:31:38.362427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:38.362401 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 05:31:39.396484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:39.396453 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:39.396984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:39.396962 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:39.397711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:39.397694 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:31:40.499913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:40.499886 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:31:49.191958 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:49.191926 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:31:59.053480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.053437 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:31:59.053955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.053707 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" podUID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" containerName="maas-api" containerID="cri-o://e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874" gracePeriod=30 Oct 30 05:31:59.274769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.273729 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:31:59.274769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.273989 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" podUID="286f49e3-ea84-473f-84d1-67c9237a56e3" containerName="payload-pre-processing" containerID="cri-o://a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2" gracePeriod=30 Oct 30 05:31:59.418366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.418335 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:31:59.485615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.485001 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:31:59.486632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.486604 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" containerName="maas-api" Oct 30 05:31:59.486632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.486633 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" containerName="maas-api" Oct 30 05:31:59.486911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.486892 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" containerName="maas-api" Oct 30 05:31:59.492754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.492730 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.502524 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.502499 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") pod \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " Oct 30 05:31:59.503071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.503048 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\" (UniqueName: \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\") pod \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " Oct 30 05:31:59.503193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.503117 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\") pod \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " Oct 30 05:31:59.514609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.503450 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\") pod \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\" (UID: \"f38f217e-7596-4095-b5ff-4e3d0ffa75fc\") " Oct 30 05:31:59.514609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.503928 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca" pod "f38f217e-7596-4095-b5ff-4e3d0ffa75fc" (UID: "f38f217e-7596-4095-b5ff-4e3d0ffa75fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:59.514609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.504986 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.514609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.509468 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b5b27e-openshift-default-dockercfg-47wp2\"" Oct 30 05:31:59.516668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.516626 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls" pod "f38f217e-7596-4095-b5ff-4e3d0ffa75fc" (UID: "f38f217e-7596-4095-b5ff-4e3d0ffa75fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:59.520306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.520283 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:31:59.520840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.520800 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85" pod "f38f217e-7596-4095-b5ff-4e3d0ffa75fc" (UID: "f38f217e-7596-4095-b5ff-4e3d0ffa75fc"). InnerVolumeSpecName "kube-api-access-nhv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:59.520942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.520835 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs" pod "f38f217e-7596-4095-b5ff-4e3d0ffa75fc" (UID: "f38f217e-7596-4095-b5ff-4e3d0ffa75fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:59.571803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.571775 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.608736 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\" (UniqueName: \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\") pod \"286f49e3-ea84-473f-84d1-67c9237a56e3\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.608976 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\") pod \"286f49e3-ea84-473f-84d1-67c9237a56e3\" (UID: \"286f49e3-ea84-473f-84d1-67c9237a56e3\") " Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.609798 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.609930 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.609964 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610059 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610114 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610137 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610159 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610211 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610280 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610502 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhv85\" (UniqueName: \"kubernetes.io/projected/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-kube-api-access-nhv85\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610519 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.610533 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38f217e-7596-4095-b5ff-4e3d0ffa75fc-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.618243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.611357 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume" pod "286f49e3-ea84-473f-84d1-67c9237a56e3" (UID: "286f49e3-ea84-473f-84d1-67c9237a56e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.702920 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn" pod "286f49e3-ea84-473f-84d1-67c9237a56e3" (UID: "286f49e3-ea84-473f-84d1-67c9237a56e3"). InnerVolumeSpecName "kube-api-access-kqjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714406 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714456 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714543 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714592 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714618 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714643 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714695 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.714744 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.715027 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.715179 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjqn\" (UniqueName: \"kubernetes.io/projected/286f49e3-ea84-473f-84d1-67c9237a56e3-kube-api-access-kqjqn\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.715195 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286f49e3-ea84-473f-84d1-67c9237a56e3-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.716035 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.719579 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.722241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.719929 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.750242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.731289 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.750242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.731598 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.773556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.770061 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.834659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.833622 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.866959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.839252 5058 generic.go:356] "Generic (PLEG): container finished" podID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" containerID="e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874" exitCode=0 Oct 30 05:31:59.866959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.839680 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" event={"ID":"f38f217e-7596-4095-b5ff-4e3d0ffa75fc","Type":"ContainerDied","Data":"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874"} Oct 30 05:31:59.866959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.840957 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" Oct 30 05:31:59.866959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.851276 5058 scope.go:122] "RemoveContainer" containerID="e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874" Oct 30 05:31:59.911088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.878871 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.911088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.885287 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\") pod \"e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:31:59.911088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.885432 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz" event={"ID":"f38f217e-7596-4095-b5ff-4e3d0ffa75fc","Type":"ContainerDied","Data":"b9bbe9d0b3b9b93a53be3d358c8d1c7b1a22fcf553d7442cdf9b86366f6cea98"} Oct 30 05:31:59.922150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.919122 5058 generic.go:356] "Generic (PLEG): container finished" podID="286f49e3-ea84-473f-84d1-67c9237a56e3" containerID="a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2" exitCode=0 Oct 30 05:31:59.922150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.919355 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" event={"ID":"286f49e3-ea84-473f-84d1-67c9237a56e3","Type":"ContainerDied","Data":"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2"} Oct 30 05:31:59.922150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.919404 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" event={"ID":"286f49e3-ea84-473f-84d1-67c9237a56e3","Type":"ContainerDied","Data":"14984ab514798d7f1d34b259c6777eee318e3d043ef63a5c5c4c7ba41b98908a"} Oct 30 05:31:59.922150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.919486 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs" Oct 30 05:31:59.948074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.948051 5058 scope.go:122] "RemoveContainer" containerID="e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874" Oct 30 05:31:59.949407 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:59.949375 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874\": container with ID starting with e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874 not found: ID does not exist" containerID="e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874" Oct 30 05:31:59.950864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.950083 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874"} err="failed to get container status \"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874\": rpc error: code = NotFound desc = could not find container \"e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874\": container with ID starting with e22bb821b99df29c71727aa9fc42049372a7032ae850255a8628c610a13ff874 not found: ID does not exist" Oct 30 05:31:59.950864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.950119 5058 scope.go:122] "RemoveContainer" containerID="a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2" Oct 30 05:31:59.965626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.965603 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:31:59.973171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.973107 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ec92c7-7b8bc8c9f9-f6tpz"] Oct 30 05:31:59.975443 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.975314 5058 scope.go:122] "RemoveContainer" containerID="a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2" Oct 30 05:31:59.976010 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:31:59.975981 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2\": container with ID starting with a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2 not found: ID does not exist" containerID="a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2" Oct 30 05:31:59.976102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:31:59.976021 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2"} err="failed to get container status \"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2\": rpc error: code = NotFound desc = could not find container \"a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2\": container with ID starting with a1044a573078434f00147592afa93316dc25823758e730b9838ce9376814f3a2 not found: ID does not exist" Oct 30 05:32:00.002765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.002732 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:32:00.007120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.007096 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs"] Oct 30 05:32:00.123701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.123670 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:32:00.517100 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:00.517070 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2340d19f_7876_4594_8113_7ae42c13e555.slice/crio-1d8b8ea2c9e3c2782fc3f7862206b61445e19dc49b3f23c2e57319890952ce1d WatchSource:0}: Error finding container 1d8b8ea2c9e3c2782fc3f7862206b61445e19dc49b3f23c2e57319890952ce1d: Status 404 returned error can't find the container with id 1d8b8ea2c9e3c2782fc3f7862206b61445e19dc49b3f23c2e57319890952ce1d Oct 30 05:32:00.517288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.517156 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:32:00.520106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.520075 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:00.520189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.520143 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:00.520257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.520197 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:00.933670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.933628 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerStarted","Data":"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617"} Oct 30 05:32:00.933882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.933690 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerStarted","Data":"1d8b8ea2c9e3c2782fc3f7862206b61445e19dc49b3f23c2e57319890952ce1d"} Oct 30 05:32:00.986160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:00.986106 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podStartSLOduration=10.986090897 podStartE2EDuration="10.986090897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:00.982382514 +0000 UTC m=+2893.951877496" watchObservedRunningTime="2026-10-30 05:32:00.986090897 +0000 UTC m=+2893.955585886" Oct 30 05:32:01.124689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.124652 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:32:01.132163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.132124 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:32:01.209293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.208569 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286f49e3-ea84-473f-84d1-67c9237a56e3" path="/var/lib/kubelet/pods/286f49e3-ea84-473f-84d1-67c9237a56e3/volumes" Oct 30 05:32:01.209481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.209377 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38f217e-7596-4095-b5ff-4e3d0ffa75fc" path="/var/lib/kubelet/pods/f38f217e-7596-4095-b5ff-4e3d0ffa75fc/volumes" Oct 30 05:32:01.945652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.945622 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:32:01.946643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:01.946619 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:32:06.471469 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.471433 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:32:06.472384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.472364 5058 cpu_manager.go:414] "containerMap: removing container" podUID="286f49e3-ea84-473f-84d1-67c9237a56e3" containerName="payload-pre-processing" Oct 30 05:32:06.472384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.472382 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286f49e3-ea84-473f-84d1-67c9237a56e3" containerName="payload-pre-processing" Oct 30 05:32:06.472547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.472536 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="286f49e3-ea84-473f-84d1-67c9237a56e3" containerName="payload-pre-processing" Oct 30 05:32:06.477804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.477785 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.497815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.497792 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5b27e\"" Oct 30 05:32:06.515251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.515211 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:32:06.558683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.558647 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.558873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.558786 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f9p\" (UniqueName: \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.661396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.661353 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f9p\" (UniqueName: \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.662475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.662446 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.675241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.663546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.706442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.706404 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f9p\" (UniqueName: \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\") pod \"payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:06.793292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:06.792894 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:07.327017 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:07.326988 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08c6510_b69a_40f3_beb5_8b4428c8e695.slice/crio-1a06911dcb419aa11be8add7d7841899432e05a742dd9a1e6520594dccb2cbc5 WatchSource:0}: Error finding container 1a06911dcb419aa11be8add7d7841899432e05a742dd9a1e6520594dccb2cbc5: Status 404 returned error can't find the container with id 1a06911dcb419aa11be8add7d7841899432e05a742dd9a1e6520594dccb2cbc5 Oct 30 05:32:07.329705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:07.329678 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:32:08.058315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.058276 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" event={"ID":"c08c6510-b69a-40f3-beb5-8b4428c8e695","Type":"ContainerStarted","Data":"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3"} Oct 30 05:32:08.058722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.058339 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" event={"ID":"c08c6510-b69a-40f3-beb5-8b4428c8e695","Type":"ContainerStarted","Data":"1a06911dcb419aa11be8add7d7841899432e05a742dd9a1e6520594dccb2cbc5"} Oct 30 05:32:08.058722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.058365 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:08.106126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.106056 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" podStartSLOduration=2.106039947 podStartE2EDuration="2.106039947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:08.103313743 +0000 UTC m=+2901.072808731" watchObservedRunningTime="2026-10-30 05:32:08.106039947 +0000 UTC m=+2901.075534934" Oct 30 05:32:08.171437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.171403 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:32:08.171666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:08.171641 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" containerID="cri-o://4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7" gracePeriod=30 Oct 30 05:32:13.492727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.492698 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:32:13.588057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.588025 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:32:13.589152 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589130 5058 cpu_manager.go:414] "containerMap: removing container" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.589250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589155 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.589250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589174 5058 cpu_manager.go:414] "containerMap: removing container" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.589250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589181 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.589455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589440 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.589496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.589460 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="914d0f0a-6044-4318-ad23-338d7e553377" containerName="istio-proxy" Oct 30 05:32:13.594086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.594055 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.613918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.611777 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-02cffa-openshift-default-dockercfg-wdscb\"" Oct 30 05:32:13.616677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.616646 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:32:13.652158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652126 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652249 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652295 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652363 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652408 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652750 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652808 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.652943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652897 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:13.652943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652917 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.653047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.652962 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\") pod \"914d0f0a-6044-4318-ad23-338d7e553377\" (UID: \"914d0f0a-6044-4318-ad23-338d7e553377\") " Oct 30 05:32:13.653573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653181 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:13.653573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653359 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:13.653573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653527 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:13.653778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653599 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:13.653986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653960 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.654080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.653990 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.654080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.654003 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.654080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.654017 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914d0f0a-6044-4318-ad23-338d7e553377-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.654080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.654030 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.655145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.655092 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:13.655145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.655114 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "kube-api-access-55vfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:13.655348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.655151 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:13.655507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.655484 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo" pod "914d0f0a-6044-4318-ad23-338d7e553377" (UID: "914d0f0a-6044-4318-ad23-338d7e553377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:13.755872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.755787 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.755926 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.755970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756044 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756079 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756114 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756148 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756235 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756412 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86zt\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.756531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756513 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vfc\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-kube-api-access-55vfc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.756579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756536 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914d0f0a-6044-4318-ad23-338d7e553377-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.756579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756547 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914d0f0a-6044-4318-ad23-338d7e553377-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.756579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.756556 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914d0f0a-6044-4318-ad23-338d7e553377-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:13.857663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857616 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.857663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857676 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.857925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857704 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.857925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857725 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.857925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857756 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.857925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857913 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86zt\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.857977 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858059 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858099 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858204 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858336 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858432 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858515 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.858691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.858669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.860259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.860195 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.860363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.860277 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.887151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.887123 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86zt\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.889116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.889095 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:13.920335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:13.920312 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:14.220342 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.220300 5058 generic.go:356] "Generic (PLEG): container finished" podID="914d0f0a-6044-4318-ad23-338d7e553377" containerID="4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7" exitCode=0 Oct 30 05:32:14.220531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.220435 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerDied","Data":"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7"} Oct 30 05:32:14.220531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.220491 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" event={"ID":"914d0f0a-6044-4318-ad23-338d7e553377","Type":"ContainerDied","Data":"68f8b014a3890c61cce968ea30d97c11350c75f049aee6a96bb3d97857313d95"} Oct 30 05:32:14.220531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.220526 5058 scope.go:122] "RemoveContainer" containerID="4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7" Oct 30 05:32:14.220704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.220668 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24" Oct 30 05:32:14.233041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.233024 5058 scope.go:122] "RemoveContainer" containerID="068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c" Oct 30 05:32:14.243416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.243395 5058 scope.go:122] "RemoveContainer" containerID="4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7" Oct 30 05:32:14.243676 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:14.243656 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7\": container with ID starting with 4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7 not found: ID does not exist" containerID="4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7" Oct 30 05:32:14.243745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.243688 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7"} err="failed to get container status \"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7\": rpc error: code = NotFound desc = could not find container \"4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7\": container with ID starting with 4c46198ef5c29841ed57e5a5daf057cde5ea10a3f44e42ef9cb52411e59d9fe7 not found: ID does not exist" Oct 30 05:32:14.243745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.243716 5058 scope.go:122] "RemoveContainer" containerID="068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c" Oct 30 05:32:14.243973 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:14.243955 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c\": container with ID starting with 068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c not found: ID does not exist" containerID="068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c" Oct 30 05:32:14.244014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.243979 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c"} err="failed to get container status \"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c\": rpc error: code = NotFound desc = could not find container \"068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c\": container with ID starting with 068e3622d8aa1085c771efc41268740394463720fb0f639326ed1223f36fd11c not found: ID does not exist" Oct 30 05:32:14.250865 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:14.250841 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d4ed5d_1090_44be_af1a_33db8320a84b.slice/crio-d90d3d20744fe558d7e8a7473170abb85d8428b70a6665228efa109865fd4ae7 WatchSource:0}: Error finding container d90d3d20744fe558d7e8a7473170abb85d8428b70a6665228efa109865fd4ae7: Status 404 returned error can't find the container with id d90d3d20744fe558d7e8a7473170abb85d8428b70a6665228efa109865fd4ae7 Oct 30 05:32:14.253966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.253941 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.254084 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.254007 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.254084 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.254042 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.256183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.256163 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:32:14.282374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.282347 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:32:14.286764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:14.286743 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24"] Oct 30 05:32:15.164237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.164197 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914d0f0a-6044-4318-ad23-338d7e553377" path="/var/lib/kubelet/pods/914d0f0a-6044-4318-ad23-338d7e553377/volumes" Oct 30 05:32:15.258946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.258912 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" event={"ID":"36d4ed5d-1090-44be-af1a-33db8320a84b","Type":"ContainerStarted","Data":"ca2d42f490aa67a5ef63e0a22bde8334871805d887d12a92f9a44d7bb9fbbcca"} Oct 30 05:32:15.259133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.258960 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" event={"ID":"36d4ed5d-1090-44be-af1a-33db8320a84b","Type":"ContainerStarted","Data":"d90d3d20744fe558d7e8a7473170abb85d8428b70a6665228efa109865fd4ae7"} Oct 30 05:32:15.920778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.920748 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:15.922003 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.921979 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 05:32:15.922098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:15.922023 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 05:32:16.921005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:16.920965 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 05:32:16.921505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:16.921034 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 05:32:17.920727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:17.920693 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 05:32:17.920896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:17.920755 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 05:32:18.920775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:18.920739 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 05:32:18.921176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:18.920810 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 05:32:19.084668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:19.084636 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:32:19.150243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:19.136476 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podStartSLOduration=9.136459365 podStartE2EDuration="9.136459365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:15.338988599 +0000 UTC m=+2908.308483593" watchObservedRunningTime="2026-10-30 05:32:19.136459365 +0000 UTC m=+2912.105954358" Oct 30 05:32:19.202479 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:19.202401 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:32:19.929341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:19.923310 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 05:32:19.929341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:19.923370 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 05:32:20.323429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.312869 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:32:20.323429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.313108 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" containerID="cri-o://56b82a7ab1b84569b56bf35f001b7dadd3100335dc2aa45b710ad000fd0dcda7" gracePeriod=30 Oct 30 05:32:20.389609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.389567 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:32:20.390711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.390664 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" containerID="cri-o://6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9" gracePeriod=30 Oct 30 05:32:20.474473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.474430 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:32:20.513771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.513742 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:32:20.926176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.926141 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:20.926646 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.926628 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:20.927301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:20.927283 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:32:25.434536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.434496 5058 generic.go:356] "Generic (PLEG): container finished" podID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerID="56b82a7ab1b84569b56bf35f001b7dadd3100335dc2aa45b710ad000fd0dcda7" exitCode=0 Oct 30 05:32:25.434904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.434573 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" event={"ID":"5d11365c-9280-4ea1-857e-218e2a1f46af","Type":"ContainerDied","Data":"56b82a7ab1b84569b56bf35f001b7dadd3100335dc2aa45b710ad000fd0dcda7"} Oct 30 05:32:25.478265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.475053 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" podUID="24bae227-08fd-42c3-abd0-4782180573fb" containerName="payload-processing" containerID="cri-o://9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712" gracePeriod=25 Oct 30 05:32:25.516454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.516382 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" podUID="6921b781-3da1-433d-a4f1-d94f74aca1a9" containerName="payload-pre-processing" containerID="cri-o://8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e" gracePeriod=25 Oct 30 05:32:25.702844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.702753 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:32:25.786641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.786610 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:32:25.787920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.787897 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" Oct 30 05:32:25.788069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.788058 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" Oct 30 05:32:25.788320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.788307 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" containerName="istio-proxy" Oct 30 05:32:25.790871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.790849 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:32:25.792804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.792788 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.812697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.812668 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:32:25.856366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.856152 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:32:25.868268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.867968 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868020 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868105 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868194 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868256 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868286 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868332 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868437 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868469 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868496 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868525 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868571 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868620 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.868941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868704 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868757 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868806 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\") pod \"5d11365c-9280-4ea1-857e-218e2a1f46af\" (UID: \"5d11365c-9280-4ea1-857e-218e2a1f46af\") " Oct 30 05:32:25.868941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868894 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.869092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.868958 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\") pod \"52de5999-f679-4863-ae87-ca4d9d7059d1\" (UID: \"52de5999-f679-4863-ae87-ca4d9d7059d1\") " Oct 30 05:32:25.869414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869389 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.869596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869579 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.869650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869616 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.869695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869655 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqpj\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.869845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869822 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.869892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.869871 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.870015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.870004 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.870059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.870033 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.870159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.870145 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.870657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.870633 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.870981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.870960 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:25.871245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.871209 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.871440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.871422 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.871605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.871589 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.878341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.878308 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:25.878539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.878518 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:25.879832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.879183 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.879832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.879390 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.879832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.879562 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.880059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.879904 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "kube-api-access-pjm5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:25.885108 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.885080 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.885391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.885366 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:32:25.885654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.885482 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:25.886860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.886829 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket" pod "5d11365c-9280-4ea1-857e-218e2a1f46af" (UID: "5d11365c-9280-4ea1-857e-218e2a1f46af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.891043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.891007 5058 cpu_manager.go:414] "containerMap: removing container" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" Oct 30 05:32:25.891043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.891038 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" Oct 30 05:32:25.891454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.891437 5058 cpu_manager.go:414] "containerMap: removing container" podUID="24bae227-08fd-42c3-abd0-4782180573fb" containerName="payload-processing" Oct 30 05:32:25.891542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.891455 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bae227-08fd-42c3-abd0-4782180573fb" containerName="payload-processing" Oct 30 05:32:25.894773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.892730 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:25.894773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.893184 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bae227-08fd-42c3-abd0-4782180573fb" containerName="payload-processing" Oct 30 05:32:25.894773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.893241 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerName="istio-proxy" Oct 30 05:32:25.908105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.907646 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:25.915779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.915734 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:25.918428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.918184 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b" pod "52de5999-f679-4863-ae87-ca4d9d7059d1" (UID: "52de5999-f679-4863-ae87-ca4d9d7059d1"). InnerVolumeSpecName "kube-api-access-55q6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:25.925684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.924498 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:32:25.925684 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.924615 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:25.936576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.936551 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:32:25.936699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.936660 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:25.938850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.938604 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:32:25.944823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.944802 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec1091-dockercfg-7hv4d\"" Oct 30 05:32:25.945059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.945043 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec1091\"" Oct 30 05:32:25.946825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.946800 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:32:25.975197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.975070 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\") pod \"24bae227-08fd-42c3-abd0-4782180573fb\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " Oct 30 05:32:25.975506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.975479 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\" (UniqueName: \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\") pod \"24bae227-08fd-42c3-abd0-4782180573fb\" (UID: \"24bae227-08fd-42c3-abd0-4782180573fb\") " Oct 30 05:32:25.976133 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.976106 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.976301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.976281 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.976437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.976421 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqpj\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.976670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.976647 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.976828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.976809 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.977035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.977018 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:25.988633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.988581 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.002940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:25.998409 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.003760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.003680 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.004578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.004531 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.004730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.004714 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de5999-f679-4863-ae87-ca4d9d7059d1-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.004851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.004837 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.004977 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.005286 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.005419 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.005614 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.005800 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11365c-9280-4ea1-857e-218e2a1f46af-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.005944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.005933 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.006053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.006023 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.006189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.006173 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55q6b\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-kube-api-access-55q6b\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.006333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.006319 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.006430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.006419 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.029030 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.006507 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.029917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.029887 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11365c-9280-4ea1-857e-218e2a1f46af-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.030103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.030087 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjm5n\" (UniqueName: \"kubernetes.io/projected/5d11365c-9280-4ea1-857e-218e2a1f46af-kube-api-access-pjm5n\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.030212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.030197 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.046319 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de5999-f679-4863-ae87-ca4d9d7059d1-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.022823 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.019071 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.019415 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.033581 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume" pod "24bae227-08fd-42c3-abd0-4782180573fb" (UID: "24bae227-08fd-42c3-abd0-4782180573fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.040415 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85" pod "24bae227-08fd-42c3-abd0-4782180573fb" (UID: "24bae227-08fd-42c3-abd0-4782180573fb"). InnerVolumeSpecName "kube-api-access-scn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.034596 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.035317 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.039743 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqpj\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.022576 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.047873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.019662 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.051748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.051718 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.121569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.121533 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:26.147708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.147682 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\" (UniqueName: \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\") pod \"6921b781-3da1-433d-a4f1-d94f74aca1a9\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " Oct 30 05:32:26.147902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.147880 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\") pod \"6921b781-3da1-433d-a4f1-d94f74aca1a9\" (UID: \"6921b781-3da1-433d-a4f1-d94f74aca1a9\") " Oct 30 05:32:26.148173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148148 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44st\" (UniqueName: \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.148377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148347 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume" pod "6921b781-3da1-433d-a4f1-d94f74aca1a9" (UID: "6921b781-3da1-433d-a4f1-d94f74aca1a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:26.148489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148458 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.148623 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148603 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.148746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148722 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nzf\" (UniqueName: \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.149034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148966 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6921b781-3da1-433d-a4f1-d94f74aca1a9-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.149034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.148997 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24bae227-08fd-42c3-abd0-4782180573fb-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.149034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.149013 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scn85\" (UniqueName: \"kubernetes.io/projected/24bae227-08fd-42c3-abd0-4782180573fb-kube-api-access-scn85\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.151502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.151476 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc" pod "6921b781-3da1-433d-a4f1-d94f74aca1a9" (UID: "6921b781-3da1-433d-a4f1-d94f74aca1a9"). InnerVolumeSpecName "kube-api-access-q2lfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:26.167006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.166970 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:32:26.168065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.168039 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6921b781-3da1-433d-a4f1-d94f74aca1a9" containerName="payload-pre-processing" Oct 30 05:32:26.168065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.168063 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6921b781-3da1-433d-a4f1-d94f74aca1a9" containerName="payload-pre-processing" Oct 30 05:32:26.168303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.168196 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6921b781-3da1-433d-a4f1-d94f74aca1a9" containerName="payload-pre-processing" Oct 30 05:32:26.175452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.175422 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.204350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.203291 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:32:26.250619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250513 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.250869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250638 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.250869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250713 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.250869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250753 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.250869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250850 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nzf\" (UniqueName: \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.251271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250916 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.251271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.250968 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44st\" (UniqueName: \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.251271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251040 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.251678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251653 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspjm\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.251783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251696 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.251871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251852 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.251939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251894 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.251996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.251939 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.252166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.252145 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2lfc\" (UniqueName: \"kubernetes.io/projected/6921b781-3da1-433d-a4f1-d94f74aca1a9-kube-api-access-q2lfc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:26.252573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.252546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.253007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.252979 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.275757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.275363 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44st\" (UniqueName: \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\") pod \"payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.276692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.276662 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nzf\" (UniqueName: \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.353509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353472 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353515 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353563 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353664 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353698 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353775 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353859 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353909 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspjm\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.353960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.353934 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.354198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.354159 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.354437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.354380 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.354719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.354693 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.354892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.354791 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.354997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.354971 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.355985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.355966 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.356464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.356445 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.374936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.374913 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspjm\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.375069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.375051 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.444400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.444346 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" event={"ID":"5d11365c-9280-4ea1-857e-218e2a1f46af","Type":"ContainerDied","Data":"1fcc87fa55e32e67b70cde33f1ceef3134516e5e9673aa1821993c67e3483f47"} Oct 30 05:32:26.444400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.444401 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg" Oct 30 05:32:26.444964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.444432 5058 scope.go:122] "RemoveContainer" containerID="56b82a7ab1b84569b56bf35f001b7dadd3100335dc2aa45b710ad000fd0dcda7" Oct 30 05:32:26.446740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.446688 5058 generic.go:356] "Generic (PLEG): container finished" podID="52de5999-f679-4863-ae87-ca4d9d7059d1" containerID="6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9" exitCode=0 Oct 30 05:32:26.450732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.447472 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" event={"ID":"52de5999-f679-4863-ae87-ca4d9d7059d1","Type":"ContainerDied","Data":"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9"} Oct 30 05:32:26.450732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.447545 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" event={"ID":"52de5999-f679-4863-ae87-ca4d9d7059d1","Type":"ContainerDied","Data":"e96a7849fd21853da96fc3794ee05c3ca587fadf518f2bf75c72cf7040ccae53"} Oct 30 05:32:26.450732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.447646 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm" Oct 30 05:32:26.453796 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:26.453770 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06864d1a_d062_4b6c_ad35_90c40a1730ed.slice/crio-1b58b2e4ddd19c3b750c89cc2bd1ec71c69d28c0c54e48de71cb9b5a02877c1c WatchSource:0}: Error finding container 1b58b2e4ddd19c3b750c89cc2bd1ec71c69d28c0c54e48de71cb9b5a02877c1c: Status 404 returned error can't find the container with id 1b58b2e4ddd19c3b750c89cc2bd1ec71c69d28c0c54e48de71cb9b5a02877c1c Oct 30 05:32:26.454425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.454397 5058 generic.go:356] "Generic (PLEG): container finished" podID="6921b781-3da1-433d-a4f1-d94f74aca1a9" containerID="8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e" exitCode=0 Oct 30 05:32:26.454553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.454534 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" event={"ID":"6921b781-3da1-433d-a4f1-d94f74aca1a9","Type":"ContainerDied","Data":"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e"} Oct 30 05:32:26.454642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.454579 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" event={"ID":"6921b781-3da1-433d-a4f1-d94f74aca1a9","Type":"ContainerDied","Data":"dbcb4cf226d94d690a603b1a32bd8884a27705b19587d14b2b1eba3fa2821ae3"} Oct 30 05:32:26.454700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.454655 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7" Oct 30 05:32:26.455606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.455553 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:32:26.457822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.457799 5058 generic.go:356] "Generic (PLEG): container finished" podID="24bae227-08fd-42c3-abd0-4782180573fb" containerID="9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712" exitCode=0 Oct 30 05:32:26.457928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.457848 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" Oct 30 05:32:26.457928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.457870 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" event={"ID":"24bae227-08fd-42c3-abd0-4782180573fb","Type":"ContainerDied","Data":"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712"} Oct 30 05:32:26.457928 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.457908 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2" event={"ID":"24bae227-08fd-42c3-abd0-4782180573fb","Type":"ContainerDied","Data":"ee269785ad916395e64fd7a9b1d8ba415f3e1329119a026f4b046728387f81fd"} Oct 30 05:32:26.458673 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.458641 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.458765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.458719 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.458765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.458759 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.479864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.479846 5058 scope.go:122] "RemoveContainer" containerID="6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9" Oct 30 05:32:26.503881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.503819 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:26.505907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.505886 5058 scope.go:122] "RemoveContainer" containerID="6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9" Oct 30 05:32:26.506287 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:26.506266 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9\": container with ID starting with 6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9 not found: ID does not exist" containerID="6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9" Oct 30 05:32:26.506373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.506296 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9"} err="failed to get container status \"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9\": rpc error: code = NotFound desc = could not find container \"6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9\": container with ID starting with 6a22cc2949b61978262403ed53ac65b0a16677cdc0991eb2a8a15a02ecc880f9 not found: ID does not exist" Oct 30 05:32:26.506373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.506313 5058 scope.go:122] "RemoveContainer" containerID="8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e" Oct 30 05:32:26.528380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.528355 5058 scope.go:122] "RemoveContainer" containerID="8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e" Oct 30 05:32:26.528738 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:26.528716 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e\": container with ID starting with 8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e not found: ID does not exist" containerID="8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e" Oct 30 05:32:26.528826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.528750 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e"} err="failed to get container status \"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e\": rpc error: code = NotFound desc = could not find container \"8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e\": container with ID starting with 8bb831b89ff7fc50068f45b3eeb98958c94ce7ac5408eaefcb9bcf2085d7142e not found: ID does not exist" Oct 30 05:32:26.528826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.528773 5058 scope.go:122] "RemoveContainer" containerID="9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712" Oct 30 05:32:26.545390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.545357 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:26.557023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.557000 5058 scope.go:122] "RemoveContainer" containerID="9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712" Oct 30 05:32:26.557552 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:26.557485 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712\": container with ID starting with 9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712 not found: ID does not exist" containerID="9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712" Oct 30 05:32:26.557552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.557525 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712"} err="failed to get container status \"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712\": rpc error: code = NotFound desc = could not find container \"9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712\": container with ID starting with 9fcf140d56a47157c764081241b24591fdd2a197c40143411718f14fa358a712 not found: ID does not exist" Oct 30 05:32:26.560248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.559399 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:26.570860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.570240 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:32:26.586275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.584781 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7"] Oct 30 05:32:26.628825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.628786 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:32:26.639871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.639841 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm"] Oct 30 05:32:26.703839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.703801 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:32:26.711304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.711024 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg"] Oct 30 05:32:26.745374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.745348 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:32:26.753845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.753808 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2"] Oct 30 05:32:26.821180 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:26.821135 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03092fc6_badb_4e1c_9114_82b110f0eec2.slice/crio-962cd3b8532e92c94f1a7117d0b3c950c8674398a659623991a31eb3f9e67f45 WatchSource:0}: Error finding container 962cd3b8532e92c94f1a7117d0b3c950c8674398a659623991a31eb3f9e67f45: Status 404 returned error can't find the container with id 962cd3b8532e92c94f1a7117d0b3c950c8674398a659623991a31eb3f9e67f45 Oct 30 05:32:26.824367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.824336 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.824459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.824412 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.824459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.824445 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:26.824760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.824738 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:32:26.916146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.916119 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:32:26.917711 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:26.917681 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f2817e_ca70_4487_b33f_6274df124564.slice/crio-380909cce646ffacfd131d0aa6cca3b677e492eeaf4b67391739fdb27c78bb57 WatchSource:0}: Error finding container 380909cce646ffacfd131d0aa6cca3b677e492eeaf4b67391739fdb27c78bb57: Status 404 returned error can't find the container with id 380909cce646ffacfd131d0aa6cca3b677e492eeaf4b67391739fdb27c78bb57 Oct 30 05:32:26.951982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:26.951962 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:32:27.166815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.166783 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24bae227-08fd-42c3-abd0-4782180573fb" path="/var/lib/kubelet/pods/24bae227-08fd-42c3-abd0-4782180573fb/volumes" Oct 30 05:32:27.167390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.167347 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52de5999-f679-4863-ae87-ca4d9d7059d1" path="/var/lib/kubelet/pods/52de5999-f679-4863-ae87-ca4d9d7059d1/volumes" Oct 30 05:32:27.167950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.167932 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d11365c-9280-4ea1-857e-218e2a1f46af" path="/var/lib/kubelet/pods/5d11365c-9280-4ea1-857e-218e2a1f46af/volumes" Oct 30 05:32:27.168715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.168698 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6921b781-3da1-433d-a4f1-d94f74aca1a9" path="/var/lib/kubelet/pods/6921b781-3da1-433d-a4f1-d94f74aca1a9/volumes" Oct 30 05:32:27.468834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.468792 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" event={"ID":"03092fc6-badb-4e1c-9114-82b110f0eec2","Type":"ContainerStarted","Data":"51ebefcfacc2e155394bc4ebba9fad21028cd6939c1bd5abac42d0d69241637d"} Oct 30 05:32:27.469378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.468857 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" event={"ID":"03092fc6-badb-4e1c-9114-82b110f0eec2","Type":"ContainerStarted","Data":"962cd3b8532e92c94f1a7117d0b3c950c8674398a659623991a31eb3f9e67f45"} Oct 30 05:32:27.471597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.471572 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" event={"ID":"68f2817e-ca70-4487-b33f-6274df124564","Type":"ContainerStarted","Data":"798f8d6ed605fdada4c9a1de795cb8476792ec10781e6d084333a1bbeacb98ed"} Oct 30 05:32:27.471715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.471618 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" event={"ID":"68f2817e-ca70-4487-b33f-6274df124564","Type":"ContainerStarted","Data":"380909cce646ffacfd131d0aa6cca3b677e492eeaf4b67391739fdb27c78bb57"} Oct 30 05:32:27.471715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.471674 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:27.473329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.473310 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" event={"ID":"c3a2571c-152e-442d-bb3b-a52ebf896e65","Type":"ContainerStarted","Data":"02495b88fd3d105e4cf56fe5760c1560045317486365d32314c62fa721ca11f3"} Oct 30 05:32:27.473419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.473344 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" event={"ID":"c3a2571c-152e-442d-bb3b-a52ebf896e65","Type":"ContainerStarted","Data":"f2eb4ad5aeb8f785e8053ca2e16b96cb168dd78aa75593d5c65183d956da3d14"} Oct 30 05:32:27.473419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.473363 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:27.475905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.475879 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" event={"ID":"06864d1a-d062-4b6c-ad35-90c40a1730ed","Type":"ContainerStarted","Data":"197d25f19c8eafa2c55748f7ef5736e7ad5739328c36d8f9b68d670c17d23b17"} Oct 30 05:32:27.476006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.475921 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" event={"ID":"06864d1a-d062-4b6c-ad35-90c40a1730ed","Type":"ContainerStarted","Data":"1b58b2e4ddd19c3b750c89cc2bd1ec71c69d28c0c54e48de71cb9b5a02877c1c"} Oct 30 05:32:27.504282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.504253 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:27.506072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.506050 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:27.506181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.506091 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:27.520620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.520561 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podStartSLOduration=17.520546444 podStartE2EDuration="17.520546444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:27.517611281 +0000 UTC m=+2920.487106268" watchObservedRunningTime="2026-10-30 05:32:27.520546444 +0000 UTC m=+2920.490041431" Oct 30 05:32:27.580596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.580520 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podStartSLOduration=17.580501349 podStartE2EDuration="17.580501349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:27.574467744 +0000 UTC m=+2920.543962733" watchObservedRunningTime="2026-10-30 05:32:27.580501349 +0000 UTC m=+2920.549996337" Oct 30 05:32:27.625624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:27.625571 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" podStartSLOduration=8.625556627 podStartE2EDuration="8.625556627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:27.619673607 +0000 UTC m=+2920.589168596" watchObservedRunningTime="2026-10-30 05:32:27.625556627 +0000 UTC m=+2920.595051614" Oct 30 05:32:28.122158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:28.122123 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:28.123989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:28.123964 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:28.124089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:28.124013 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:28.505107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:28.505074 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:28.505509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:28.505120 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:29.122743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.122695 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:29.122969 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.122754 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:29.294246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.293963 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" podStartSLOduration=10.293943256 podStartE2EDuration="10.293943256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:27.666434546 +0000 UTC m=+2920.635929535" watchObservedRunningTime="2026-10-30 05:32:29.293943256 +0000 UTC m=+2922.263438244" Oct 30 05:32:29.298072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.296481 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:32:29.298072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.296724 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" containerID="cri-o://09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869" gracePeriod=30 Oct 30 05:32:29.711783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.711734 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:32:29.712626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.712592 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" podUID="ad10ddec-4609-4a69-9451-05eb371dc929" containerName="maas-api" containerID="cri-o://c3f6f04435f75cd915de0d530941fc1e46ab953c3b87623a4029311944caa706" gracePeriod=30 Oct 30 05:32:29.773177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.772865 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:29.773177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.772927 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:29.774061 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.774025 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:32:29.774282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:29.774258 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:32:30.260275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.246356 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:30.260275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.246415 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:30.504763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.504710 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:30.505045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.504993 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:30.522365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.520061 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:32:30.915138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.913472 5058 generic.go:356] "Generic (PLEG): container finished" podID="ad10ddec-4609-4a69-9451-05eb371dc929" containerID="c3f6f04435f75cd915de0d530941fc1e46ab953c3b87623a4029311944caa706" exitCode=0 Oct 30 05:32:30.915138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:30.913573 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" event={"ID":"ad10ddec-4609-4a69-9451-05eb371dc929","Type":"ContainerDied","Data":"c3f6f04435f75cd915de0d530941fc1e46ab953c3b87623a4029311944caa706"} Oct 30 05:32:31.139301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.122423 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:31.139301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.122493 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:31.292612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.292587 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:32:31.376242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.359449 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\" (UniqueName: \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\") pod \"ad10ddec-4609-4a69-9451-05eb371dc929\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " Oct 30 05:32:31.376242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.359812 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\") pod \"ad10ddec-4609-4a69-9451-05eb371dc929\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " Oct 30 05:32:31.376242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.359857 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") pod \"ad10ddec-4609-4a69-9451-05eb371dc929\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " Oct 30 05:32:31.376242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.360001 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\") pod \"ad10ddec-4609-4a69-9451-05eb371dc929\" (UID: \"ad10ddec-4609-4a69-9451-05eb371dc929\") " Oct 30 05:32:31.392330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.392197 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs" pod "ad10ddec-4609-4a69-9451-05eb371dc929" (UID: "ad10ddec-4609-4a69-9451-05eb371dc929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:31.393016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.392987 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca" pod "ad10ddec-4609-4a69-9451-05eb371dc929" (UID: "ad10ddec-4609-4a69-9451-05eb371dc929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:31.397253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.394542 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad10ddec-4609-4a69-9451-05eb371dc929-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.397253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.394571 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.402818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.399679 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls" pod "ad10ddec-4609-4a69-9451-05eb371dc929" (UID: "ad10ddec-4609-4a69-9451-05eb371dc929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:31.408479 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.408378 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:32:31.412887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.411359 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ad10ddec-4609-4a69-9451-05eb371dc929" containerName="maas-api" Oct 30 05:32:31.412887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.411379 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad10ddec-4609-4a69-9451-05eb371dc929" containerName="maas-api" Oct 30 05:32:31.412887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.411590 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad10ddec-4609-4a69-9451-05eb371dc929" containerName="maas-api" Oct 30 05:32:31.415989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.415385 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht" pod "ad10ddec-4609-4a69-9451-05eb371dc929" (UID: "ad10ddec-4609-4a69-9451-05eb371dc929"). InnerVolumeSpecName "kube-api-access-9b9ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:31.417619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.417601 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:32:31.417697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.417682 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504089 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:31.504323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504137 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:31.504323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504232 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504288 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4lbl\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504466 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504519 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504653 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504739 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504777 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504818 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.504975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.504957 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.505368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.505348 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad10ddec-4609-4a69-9451-05eb371dc929-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.505417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.505376 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9ht\" (UniqueName: \"kubernetes.io/projected/ad10ddec-4609-4a69-9451-05eb371dc929-kube-api-access-9b9ht\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.606621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606583 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.606795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606628 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.606795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606664 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.606795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606721 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.606944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606926 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.606984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.606969 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4lbl\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607317 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607358 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607369 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607487 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607445 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607591 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.607913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.607892 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.609463 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.609438 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.609734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.609718 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.669290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.668382 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4lbl\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.669290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.668874 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.811627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.811582 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:31.937294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.936951 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" event={"ID":"ad10ddec-4609-4a69-9451-05eb371dc929","Type":"ContainerDied","Data":"16adcdbc92f35c7508d2267ca4dce754c3ced2ca7af150cefbbf1030542724b9"} Oct 30 05:32:31.937294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.937025 5058 scope.go:122] "RemoveContainer" containerID="c3f6f04435f75cd915de0d530941fc1e46ab953c3b87623a4029311944caa706" Oct 30 05:32:31.937294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:31.937195 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979" Oct 30 05:32:32.101373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.101333 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:32:32.106809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.106775 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-097d90-555f7969c5-z6979"] Oct 30 05:32:32.144240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.122857 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:32.144240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.122913 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:32.279481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.277957 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:32:32.285334 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:32.285302 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44504719_7e7d_43af_961c_f5966c48c968.slice/crio-37c4a32a8ad3bd15e7683171135680fc978d5b52ecc67b11cee1a0cfdaae4f0d WatchSource:0}: Error finding container 37c4a32a8ad3bd15e7683171135680fc978d5b52ecc67b11cee1a0cfdaae4f0d: Status 404 returned error can't find the container with id 37c4a32a8ad3bd15e7683171135680fc978d5b52ecc67b11cee1a0cfdaae4f0d Oct 30 05:32:32.288425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.288395 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:32.288505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.288460 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:32.288505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.288491 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:32.506258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.504936 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:32.506258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.504997 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:32.988945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.988435 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" event={"ID":"44504719-7e7d-43af-961c-f5966c48c968","Type":"ContainerStarted","Data":"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc"} Oct 30 05:32:32.988945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:32.988505 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" event={"ID":"44504719-7e7d-43af-961c-f5966c48c968","Type":"ContainerStarted","Data":"37c4a32a8ad3bd15e7683171135680fc978d5b52ecc67b11cee1a0cfdaae4f0d"} Oct 30 05:32:33.059296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.058279 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podStartSLOduration=121.058257377 podStartE2EDuration="2m1.058257377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:33.041798676 +0000 UTC m=+2926.011293656" watchObservedRunningTime="2026-10-30 05:32:33.058257377 +0000 UTC m=+2926.027752366" Oct 30 05:32:33.122488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.122455 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 05:32:33.122674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.122523 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 05:32:33.190379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.190348 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad10ddec-4609-4a69-9451-05eb371dc929" path="/var/lib/kubelet/pods/ad10ddec-4609-4a69-9451-05eb371dc929/volumes" Oct 30 05:32:33.504946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.504909 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 05:32:33.505235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.505188 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 05:32:33.811971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.811787 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:33.814380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.814341 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:33.814563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:33.814412 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:34.142758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.140412 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:34.142758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.142295 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:34.162297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.160867 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:32:34.511790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.510885 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:34.511790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.511631 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:34.512718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.512694 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:32:34.702904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.702870 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:32:34.714356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.714325 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.714629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.714592 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.715163 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.715694 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.715730 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.715774 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.716043 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.716113 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.716230 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.716341 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.716579 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.717274 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.717297 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.717310 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2077227-c757-45f6-b0aa-3f901ba4988a-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.718590 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.722587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.720172 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:34.723384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.723368 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.732275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.724212 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "kube-api-access-mlbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:34.767853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.767801 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:34.792596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.787870 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:32:34.792596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.789042 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:34.792596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.789062 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:34.792596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.789207 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:34.799542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.799514 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.806818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.806789 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:32:34.812291 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.812258 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:34.812417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.812328 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:34.819042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819009 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\") pod \"f2077227-c757-45f6-b0aa-3f901ba4988a\" (UID: \"f2077227-c757-45f6-b0aa-3f901ba4988a\") " Oct 30 05:32:34.819367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819341 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data" pod "f2077227-c757-45f6-b0aa-3f901ba4988a" (UID: "f2077227-c757-45f6-b0aa-3f901ba4988a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.819467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819438 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.819544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819481 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.819544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819514 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.819745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819725 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.819869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.819845 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.820040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.820026 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.820389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.820345 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.820583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.820561 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwx8\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.820774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.820736 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.820997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.820980 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbbm\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-kube-api-access-mlbbm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.821100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.821087 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.821186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.821175 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.821305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.821289 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.821396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.821385 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.821483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.821472 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2077227-c757-45f6-b0aa-3f901ba4988a-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.922734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.922698 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.922937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.922750 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwx8\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.922937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.922785 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.922937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.922817 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.922937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.922837 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923099 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923118 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923160 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923308 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923390 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923419 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.923554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923538 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.924040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923706 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.924040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.923967 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.949294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.934821 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.949294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.935444 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.952268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.952237 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwx8\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:34.953324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:34.953290 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:35.009896 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.009863 5058 generic.go:356] "Generic (PLEG): container finished" podID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerID="09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869" exitCode=0 Oct 30 05:32:35.010827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.010809 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" Oct 30 05:32:35.011785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.011561 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerDied","Data":"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869"} Oct 30 05:32:35.011785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.011631 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992" event={"ID":"f2077227-c757-45f6-b0aa-3f901ba4988a","Type":"ContainerDied","Data":"175625dafbfac70d9c2d6ebb884fda93c66c024fb85cd65cc2eb136a51c5eff5"} Oct 30 05:32:35.011785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.011663 5058 scope.go:122] "RemoveContainer" containerID="09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869" Oct 30 05:32:35.057205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.057181 5058 scope.go:122] "RemoveContainer" containerID="7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f" Oct 30 05:32:35.069609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.069589 5058 scope.go:122] "RemoveContainer" containerID="09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869" Oct 30 05:32:35.069917 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:35.069890 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869\": container with ID starting with 09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869 not found: ID does not exist" containerID="09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869" Oct 30 05:32:35.070010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.069927 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869"} err="failed to get container status \"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869\": rpc error: code = NotFound desc = could not find container \"09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869\": container with ID starting with 09495f8266605a8913c45f47713c5d79ac44bee71c1a1337fcf722cfaa825869 not found: ID does not exist" Oct 30 05:32:35.070010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.069945 5058 scope.go:122] "RemoveContainer" containerID="7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f" Oct 30 05:32:35.070309 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:35.070262 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f\": container with ID starting with 7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f not found: ID does not exist" containerID="7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f" Oct 30 05:32:35.070382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.070297 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f"} err="failed to get container status \"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f\": rpc error: code = NotFound desc = could not find container \"7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f\": container with ID starting with 7dd884dd756b0035eb50c5ac40ac1c7009940a86c73ad40ca0772dd43ef76f3f not found: ID does not exist" Oct 30 05:32:35.095747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.095707 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:32:35.100326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.100293 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992"] Oct 30 05:32:35.124079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.124050 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:35.166537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.166504 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" path="/var/lib/kubelet/pods/f2077227-c757-45f6-b0aa-3f901ba4988a/volumes" Oct 30 05:32:35.476106 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:35.476066 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945b59f2_18c2_4bfb_8050_6f0e5dc8a1d6.slice/crio-9b171de2b0c7c79c593bb5f88f976f5553e13350c9e1b6c1e0693e4a952dae19 WatchSource:0}: Error finding container 9b171de2b0c7c79c593bb5f88f976f5553e13350c9e1b6c1e0693e4a952dae19: Status 404 returned error can't find the container with id 9b171de2b0c7c79c593bb5f88f976f5553e13350c9e1b6c1e0693e4a952dae19 Oct 30 05:32:35.478186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.478160 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:32:35.479010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.478979 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.479055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.479047 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.479092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.479077 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.812232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.812137 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:35.812232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:35.812199 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:36.021990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.021947 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" event={"ID":"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6","Type":"ContainerStarted","Data":"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e"} Oct 30 05:32:36.022172 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.022020 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" event={"ID":"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6","Type":"ContainerStarted","Data":"9b171de2b0c7c79c593bb5f88f976f5553e13350c9e1b6c1e0693e4a952dae19"} Oct 30 05:32:36.081347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.081244 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podStartSLOduration=131.081208227 podStartE2EDuration="2m11.081208227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:36.080548682 +0000 UTC m=+2929.050043671" watchObservedRunningTime="2026-10-30 05:32:36.081208227 +0000 UTC m=+2929.050703215" Oct 30 05:32:36.155468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.155375 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:36.155812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.155780 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:36.155963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.155840 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:36.812538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.812503 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:36.812926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:36.812564 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:37.145301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:37.133597 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:37.145301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:37.133661 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:37.812141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:37.812099 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:37.812345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:37.812166 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:38.174455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.174414 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:38.175075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.175041 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:38.473392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.467145 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:32:38.551242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.541182 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" containerID="cri-o://49439db0b272e97d53f9dd7cfb86fab39eb364940daff4aff691cae9454ef08f" gracePeriod=30 Oct 30 05:32:38.800304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.791775 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:32:38.800304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.794888 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:32:38.819249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.816561 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 05:32:38.819249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.816621 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 05:32:38.916318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.916285 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b"] Oct 30 05:32:38.917786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.917759 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:38.917938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.917926 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:38.918431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.918411 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2077227-c757-45f6-b0aa-3f901ba4988a" containerName="istio-proxy" Oct 30 05:32:38.945714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.924450 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:38.979243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.973506 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:32:38.979243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.973664 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b"] Oct 30 05:32:38.979243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:38.973737 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:32:39.094933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.094847 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6pr2\" (UniqueName: \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.095107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.095039 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.095180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.095111 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.152287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.152250 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:39.152588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.152554 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:39.200894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.200850 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6pr2\" (UniqueName: \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.201606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.201578 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.201804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.201787 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.202973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.202926 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.206098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.206081 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.222306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.222284 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6pr2\" (UniqueName: \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.300368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.300336 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:39.723097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.722539 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b"] Oct 30 05:32:39.894538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.894502 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:39.895533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.895331 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:39.896426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:39.896402 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:32:40.169603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:40.169566 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:40.169798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:40.169622 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:40.218079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:40.218046 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" event={"ID":"65444801-f92b-47d5-8b44-ed72c4e8a1c7","Type":"ContainerStarted","Data":"c214abaf9bf77f9d0d3f09988b1b8c4328bb50ca1da8629dc8ad598e1857bc33"} Oct 30 05:32:41.149244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:41.148657 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:41.149244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:41.148723 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:42.132709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:42.132666 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 05:32:42.133316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:42.133272 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 05:32:43.130992 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:43.130965 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:43.131376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:43.131355 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:43.132039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:43.132020 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:32:43.280211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:43.280175 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" event={"ID":"65444801-f92b-47d5-8b44-ed72c4e8a1c7","Type":"ContainerStarted","Data":"4f45c6a10f7c87b38d4c7a0af487466883f5f86df6ba6f8c3b68fc5759bce2b9"} Oct 30 05:32:43.339636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:43.339590 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" podStartSLOduration=2.4114147839999998 podStartE2EDuration="5.339574703s" totalImagesPullingTime="2.928159919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:43.337542388 +0000 UTC m=+2936.307037365" watchObservedRunningTime="2026-10-30 05:32:43.339574703 +0000 UTC m=+2936.309069690" Oct 30 05:32:44.294240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.293324 5058 generic.go:356] "Generic (PLEG): container finished" podID="65444801-f92b-47d5-8b44-ed72c4e8a1c7" containerID="4f45c6a10f7c87b38d4c7a0af487466883f5f86df6ba6f8c3b68fc5759bce2b9" exitCode=0 Oct 30 05:32:44.294240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.293441 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" event={"ID":"65444801-f92b-47d5-8b44-ed72c4e8a1c7","Type":"ContainerDied","Data":"4f45c6a10f7c87b38d4c7a0af487466883f5f86df6ba6f8c3b68fc5759bce2b9"} Oct 30 05:32:44.296462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.296428 5058 generic.go:356] "Generic (PLEG): container finished" podID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerID="49439db0b272e97d53f9dd7cfb86fab39eb364940daff4aff691cae9454ef08f" exitCode=0 Oct 30 05:32:44.296599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.296507 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerDied","Data":"49439db0b272e97d53f9dd7cfb86fab39eb364940daff4aff691cae9454ef08f"} Oct 30 05:32:44.296662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.296621 5058 scope.go:122] "RemoveContainer" containerID="eade0cae3c9157218e8898a5af77bb04f01f00e0937ec2ea3285db199c5aaddd" Oct 30 05:32:44.676687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.676655 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:32:44.745579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.745537 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:32:44.746467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.746446 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:44.746467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.746465 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:44.746704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.746688 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:44.746766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.746712 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:44.750472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.750454 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.772302 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.772274 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:32:44.873782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873697 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.873782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873739 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.873782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873780 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.873995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873824 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.873995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873890 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.873995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.873964 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.874123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874065 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.874177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874120 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:44.874265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874167 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:44.874265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874235 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.874388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874365 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\") pod \"4a9296f7-7aed-4117-9407-d5a43216af6a\" (UID: \"4a9296f7-7aed-4117-9407-d5a43216af6a\") " Oct 30 05:32:44.874446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874365 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:44.874559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874536 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:44.874664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874566 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:44.874751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874655 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.874751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874731 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.874864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.874757 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875022 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875048 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875073 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875252 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875288 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875364 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfg4q\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.875594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875574 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.875652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875599 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.875652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875613 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.875652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875627 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.875652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.875641 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a9296f7-7aed-4117-9407-d5a43216af6a-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.876357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.876334 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:44.876572 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.876551 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:44.876632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.876568 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:44.876632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.876585 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9" pod "4a9296f7-7aed-4117-9407-d5a43216af6a" (UID: "4a9296f7-7aed-4117-9407-d5a43216af6a"). InnerVolumeSpecName "kube-api-access-d9pn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:44.976786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.976745 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.976786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.976787 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.976917 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977057 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977103 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977137 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfg4q\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977206 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977318 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977346 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977494 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977499 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.977599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977544 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.977599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977560 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pn9\" (UniqueName: \"kubernetes.io/projected/4a9296f7-7aed-4117-9407-d5a43216af6a-kube-api-access-d9pn9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.977599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977575 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a9296f7-7aed-4117-9407-d5a43216af6a-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:44.977812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977723 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.977950 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.977829 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.978073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.978050 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.978203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.978184 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.979753 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.979725 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.979871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.979844 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:44.999428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:44.999405 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:45.000090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.000074 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfg4q\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:45.076778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.076725 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:45.305791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.305753 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" event={"ID":"4a9296f7-7aed-4117-9407-d5a43216af6a","Type":"ContainerDied","Data":"25e5ac2e409ce7748137cb6fc7460bb20000a7c4f30943338fbedbdab8541ed6"} Oct 30 05:32:45.306283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.305824 5058 scope.go:122] "RemoveContainer" containerID="49439db0b272e97d53f9dd7cfb86fab39eb364940daff4aff691cae9454ef08f" Oct 30 05:32:45.306283 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.305764 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms" Oct 30 05:32:45.381027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.378362 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:32:45.393568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.393211 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms"] Oct 30 05:32:45.429087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.429059 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:32:45.431206 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:45.431181 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498528ef_a903_48ac_8a61_4384457579ec.slice/crio-d328ca4d4e925fbc4ff2fbffaf6cdb8d5b57a8adbdf9777947c1911c5590f1f8 WatchSource:0}: Error finding container d328ca4d4e925fbc4ff2fbffaf6cdb8d5b57a8adbdf9777947c1911c5590f1f8: Status 404 returned error can't find the container with id d328ca4d4e925fbc4ff2fbffaf6cdb8d5b57a8adbdf9777947c1911c5590f1f8 Oct 30 05:32:45.434955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.434913 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:45.435089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.435007 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:45.435089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.435053 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:45.466341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.466316 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:45.487744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.487721 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\" (UniqueName: \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\") pod \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " Oct 30 05:32:45.487919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.487902 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\") pod \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " Oct 30 05:32:45.488071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.488044 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\") pod \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\" (UID: \"65444801-f92b-47d5-8b44-ed72c4e8a1c7\") " Oct 30 05:32:45.488707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.488666 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca" pod "65444801-f92b-47d5-8b44-ed72c4e8a1c7" (UID: "65444801-f92b-47d5-8b44-ed72c4e8a1c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:45.490178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.490147 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca" pod "65444801-f92b-47d5-8b44-ed72c4e8a1c7" (UID: "65444801-f92b-47d5-8b44-ed72c4e8a1c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:45.491416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.491386 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2" pod "65444801-f92b-47d5-8b44-ed72c4e8a1c7" (UID: "65444801-f92b-47d5-8b44-ed72c4e8a1c7"). InnerVolumeSpecName "kube-api-access-s6pr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:45.493903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.493874 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-openshift-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:45.494005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.493907 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6pr2\" (UniqueName: \"kubernetes.io/projected/65444801-f92b-47d5-8b44-ed72c4e8a1c7-kube-api-access-s6pr2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:45.494005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:45.493922 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65444801-f92b-47d5-8b44-ed72c4e8a1c7-maas-api-xks-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:46.317284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.317189 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" event={"ID":"498528ef-a903-48ac-8a61-4384457579ec","Type":"ContainerStarted","Data":"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb"} Oct 30 05:32:46.317709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.317289 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" event={"ID":"498528ef-a903-48ac-8a61-4384457579ec","Type":"ContainerStarted","Data":"d328ca4d4e925fbc4ff2fbffaf6cdb8d5b57a8adbdf9777947c1911c5590f1f8"} Oct 30 05:32:46.318824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.318802 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" Oct 30 05:32:46.318951 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.318835 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b" event={"ID":"65444801-f92b-47d5-8b44-ed72c4e8a1c7","Type":"ContainerDied","Data":"c214abaf9bf77f9d0d3f09988b1b8c4328bb50ca1da8629dc8ad598e1857bc33"} Oct 30 05:32:46.318951 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.318876 5058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c214abaf9bf77f9d0d3f09988b1b8c4328bb50ca1da8629dc8ad598e1857bc33" Oct 30 05:32:46.400975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:46.400925 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podStartSLOduration=11.40091247 podStartE2EDuration="11.40091247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:46.397933882 +0000 UTC m=+2939.367428870" watchObservedRunningTime="2026-10-30 05:32:46.40091247 +0000 UTC m=+2939.370407529" Oct 30 05:32:47.078885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:47.078848 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:47.083301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:47.081558 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 05:32:47.083301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:47.081623 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 05:32:47.167726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:47.167693 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" path="/var/lib/kubelet/pods/4a9296f7-7aed-4117-9407-d5a43216af6a/volumes" Oct 30 05:32:48.077542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:48.077498 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 05:32:48.078022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:48.077579 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 05:32:49.077327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:49.077287 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 05:32:49.077521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:49.077352 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 05:32:49.192726 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:49.192691 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:32:49.397075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:49.397045 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:32:50.089808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:50.088693 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 05:32:50.089808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:50.088763 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 05:32:51.077980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:51.077946 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 05:32:51.078425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:51.078001 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 05:32:52.083402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:52.083369 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:52.083905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:52.083882 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:52.084698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:52.084675 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:32:54.397345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.397287 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" podUID="af53fc3a-8d64-482e-9afd-93e12cebe055" containerName="payload-pre-processing" containerID="cri-o://8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10" gracePeriod=25 Oct 30 05:32:54.759377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.759345 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:32:54.926897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.926864 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\") pod \"af53fc3a-8d64-482e-9afd-93e12cebe055\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " Oct 30 05:32:54.927150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.927134 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\" (UniqueName: \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\") pod \"af53fc3a-8d64-482e-9afd-93e12cebe055\" (UID: \"af53fc3a-8d64-482e-9afd-93e12cebe055\") " Oct 30 05:32:54.927276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.927210 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume" pod "af53fc3a-8d64-482e-9afd-93e12cebe055" (UID: "af53fc3a-8d64-482e-9afd-93e12cebe055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:54.927895 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.927872 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af53fc3a-8d64-482e-9afd-93e12cebe055-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:54.929422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:54.929398 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf" pod "af53fc3a-8d64-482e-9afd-93e12cebe055" (UID: "af53fc3a-8d64-482e-9afd-93e12cebe055"). InnerVolumeSpecName "kube-api-access-49smf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:55.045804 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.045709 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49smf\" (UniqueName: \"kubernetes.io/projected/af53fc3a-8d64-482e-9afd-93e12cebe055-kube-api-access-49smf\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:55.419415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.419377 5058 generic.go:356] "Generic (PLEG): container finished" podID="af53fc3a-8d64-482e-9afd-93e12cebe055" containerID="8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10" exitCode=0 Oct 30 05:32:55.419888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.419440 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" Oct 30 05:32:55.419888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.419464 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" event={"ID":"af53fc3a-8d64-482e-9afd-93e12cebe055","Type":"ContainerDied","Data":"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10"} Oct 30 05:32:55.419888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.419539 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf" event={"ID":"af53fc3a-8d64-482e-9afd-93e12cebe055","Type":"ContainerDied","Data":"aa16e47f14557444ad6a409f2905a0c52a59d4a1927af4e61f8c7b912c025331"} Oct 30 05:32:55.419888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.419568 5058 scope.go:122] "RemoveContainer" containerID="8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10" Oct 30 05:32:55.430843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.430824 5058 scope.go:122] "RemoveContainer" containerID="8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10" Oct 30 05:32:55.431140 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:55.431118 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10\": container with ID starting with 8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10 not found: ID does not exist" containerID="8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10" Oct 30 05:32:55.431271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.431147 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10"} err="failed to get container status \"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10\": rpc error: code = NotFound desc = could not find container \"8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10\": container with ID starting with 8872320fc6718df1a94929bcf680dc4b09c33e5842521b0d8842d745fd123e10 not found: ID does not exist" Oct 30 05:32:55.482705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.482672 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:32:55.497392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:55.497358 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf"] Oct 30 05:32:56.467544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:56.467508 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:32:56.468169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:56.467804 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerName="maas-api" containerID="cri-o://0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1" gracePeriod=30 Oct 30 05:32:56.599144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:56.599110 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.105:8443/health\": dial tcp 10.132.0.105:8443: connect: connection refused" Oct 30 05:32:56.910948 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:56.910917 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:32:57.032243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.024382 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\" (UniqueName: \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\") pod \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " Oct 30 05:32:57.032243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.024612 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\") pod \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " Oct 30 05:32:57.032243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.024825 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\") pod \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " Oct 30 05:32:57.032243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.024936 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\") pod \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\" (UID: \"569e14f1-b49c-4832-80d9-f7c5a2c48fad\") " Oct 30 05:32:57.072067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.072010 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca" pod "569e14f1-b49c-4832-80d9-f7c5a2c48fad" (UID: "569e14f1-b49c-4832-80d9-f7c5a2c48fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:57.135284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.124681 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs" pod "569e14f1-b49c-4832-80d9-f7c5a2c48fad" (UID: "569e14f1-b49c-4832-80d9-f7c5a2c48fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:57.135284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.128032 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:57.135284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.128060 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569e14f1-b49c-4832-80d9-f7c5a2c48fad-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:57.255189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.255153 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls" pod "569e14f1-b49c-4832-80d9-f7c5a2c48fad" (UID: "569e14f1-b49c-4832-80d9-f7c5a2c48fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:57.272911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.272845 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59" pod "569e14f1-b49c-4832-80d9-f7c5a2c48fad" (UID: "569e14f1-b49c-4832-80d9-f7c5a2c48fad"). InnerVolumeSpecName "kube-api-access-mrh59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:57.303106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.302965 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af53fc3a-8d64-482e-9afd-93e12cebe055" path="/var/lib/kubelet/pods/af53fc3a-8d64-482e-9afd-93e12cebe055/volumes" Oct 30 05:32:57.304344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.303785 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.304920 5058 cpu_manager.go:414] "containerMap: removing container" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerName="maas-api" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.304939 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerName="maas-api" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.304989 5058 cpu_manager.go:414] "containerMap: removing container" podUID="65444801-f92b-47d5-8b44-ed72c4e8a1c7" containerName="revoke-keys" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.304997 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65444801-f92b-47d5-8b44-ed72c4e8a1c7" containerName="revoke-keys" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.305015 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.305026 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9296f7-7aed-4117-9407-d5a43216af6a" containerName="istio-proxy" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.305043 5058 cpu_manager.go:414] "containerMap: removing container" podUID="af53fc3a-8d64-482e-9afd-93e12cebe055" containerName="payload-pre-processing" Oct 30 05:32:57.305207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.305051 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af53fc3a-8d64-482e-9afd-93e12cebe055" containerName="payload-pre-processing" Oct 30 05:32:57.307966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.306092 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerName="maas-api" Oct 30 05:32:57.307966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.306127 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="65444801-f92b-47d5-8b44-ed72c4e8a1c7" containerName="revoke-keys" Oct 30 05:32:57.307966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.306145 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="af53fc3a-8d64-482e-9afd-93e12cebe055" containerName="payload-pre-processing" Oct 30 05:32:57.311548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.311525 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:32:57.311674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.311636 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.336547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.336517 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrh59\" (UniqueName: \"kubernetes.io/projected/569e14f1-b49c-4832-80d9-f7c5a2c48fad-kube-api-access-mrh59\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:57.336547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.336545 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569e14f1-b49c-4832-80d9-f7c5a2c48fad-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:32:57.438178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438143 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438438 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438261 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438438 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438417 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438473 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nm47\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438557 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438749 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438826 5058 generic.go:356] "Generic (PLEG): container finished" podID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" containerID="0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1" exitCode=0 Oct 30 05:32:57.438914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438867 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.438914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438897 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" Oct 30 05:32:57.438914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438899 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" event={"ID":"569e14f1-b49c-4832-80d9-f7c5a2c48fad","Type":"ContainerDied","Data":"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1"} Oct 30 05:32:57.439063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438950 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775" event={"ID":"569e14f1-b49c-4832-80d9-f7c5a2c48fad","Type":"ContainerDied","Data":"9ac7af78a73b15ec6ca5a654aeb80b7ba83ee797207ce1b6493c7e5617c9fc46"} Oct 30 05:32:57.439063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438982 5058 scope.go:122] "RemoveContainer" containerID="0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1" Oct 30 05:32:57.439063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.438997 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.439278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.439196 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.451239 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.451201 5058 scope.go:122] "RemoveContainer" containerID="0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1" Oct 30 05:32:57.451556 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:32:57.451532 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1\": container with ID starting with 0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1 not found: ID does not exist" containerID="0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1" Oct 30 05:32:57.451608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.451567 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1"} err="failed to get container status \"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1\": rpc error: code = NotFound desc = could not find container \"0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1\": container with ID starting with 0feca55f3ae037243232d9b36a23a876fd2cd73705bfea1bd0276d3268211ae1 not found: ID does not exist" Oct 30 05:32:57.514114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.514076 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:32:57.537589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.537553 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e15c99-84bcdd5d95-mc775"] Oct 30 05:32:57.540886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.540857 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.540926 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.540993 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541025 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541128 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541178 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541304 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541356 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nm47\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541405 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.541493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541438 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.542814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.541847 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.542814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.542753 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.543016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.542929 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.543129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.543101 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.544545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.544521 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.545474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.545453 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.570965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.570933 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.571908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.571883 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nm47\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.634911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.634875 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:57.662286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.662244 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b"] Oct 30 05:32:57.666972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.666947 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e15c99-tzl6b"] Oct 30 05:32:57.963978 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:32:57.963945 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb5320a_45ef_4161_92a2_c9dabe83bd27.slice/crio-a78f040c134c2c45c8b1b9ab0fc472b7faf658a8c042a711ee20f49f852a6bc3 WatchSource:0}: Error finding container a78f040c134c2c45c8b1b9ab0fc472b7faf658a8c042a711ee20f49f852a6bc3: Status 404 returned error can't find the container with id a78f040c134c2c45c8b1b9ab0fc472b7faf658a8c042a711ee20f49f852a6bc3 Oct 30 05:32:57.966621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.966591 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:32:57.967213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.967183 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:57.967335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.967317 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:57.967379 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:57.967361 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:58.452771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:58.452732 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" event={"ID":"5eb5320a-45ef-4161-92a2-c9dabe83bd27","Type":"ContainerStarted","Data":"0ec075d6944e9c491283b76001ed9505f11d5343b2db71006080d79ebb89ffda"} Oct 30 05:32:58.452960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:58.452799 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" event={"ID":"5eb5320a-45ef-4161-92a2-c9dabe83bd27","Type":"ContainerStarted","Data":"a78f040c134c2c45c8b1b9ab0fc472b7faf658a8c042a711ee20f49f852a6bc3"} Oct 30 05:32:58.545782 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:58.545714 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podStartSLOduration=3.545691714 podStartE2EDuration="3.545691714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:58.524741807 +0000 UTC m=+2951.494236820" watchObservedRunningTime="2026-10-30 05:32:58.545691714 +0000 UTC m=+2951.515186702" Oct 30 05:32:58.635723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:58.635695 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:32:59.212733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:59.212689 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569e14f1-b49c-4832-80d9-f7c5a2c48fad" path="/var/lib/kubelet/pods/569e14f1-b49c-4832-80d9-f7c5a2c48fad/volumes" Oct 30 05:32:59.213436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:59.213410 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65444801-f92b-47d5-8b44-ed72c4e8a1c7" path="/var/lib/kubelet/pods/65444801-f92b-47d5-8b44-ed72c4e8a1c7/volumes" Oct 30 05:32:59.654904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:59.654865 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:32:59.655548 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:32:59.655511 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:00.651042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:00.648435 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:33:00.651042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:00.648501 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:01.649751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:01.649700 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:33:01.650329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:01.649778 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:02.649784 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:02.649737 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:33:02.650426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:02.649812 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:03.644700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:03.644658 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:33:03.645024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:03.644996 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:04.645176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:04.645134 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 05:33:04.688396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:04.688354 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 05:33:05.685287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:05.671513 5058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 30 05:33:05.698298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:05.696961 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.893375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:05.892341 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.000426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.000324 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.188058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.188005 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.552262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.544523 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:33:06.564255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.563460 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:33:06.584947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.579460 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:33:06.676246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.668497 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.722009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.721931 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.752269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.742852 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:33:06.968518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:06.968486 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:33:09.299259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.299180 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8864-f40e45"] Oct 30 05:33:09.306156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.306132 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:09.340118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.340089 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\"" Oct 30 05:33:09.342049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.342009 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8864-f40e45"] Oct 30 05:33:09.343377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.343347 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw92l\" (UniqueName: \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\") pod \"test-curl-8864-f40e45\" (UID: \"4ca5f0b9-6002-4a37-9d66-d4359e65bc86\") " pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:09.483241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.451621 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw92l\" (UniqueName: \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\") pod \"test-curl-8864-f40e45\" (UID: \"4ca5f0b9-6002-4a37-9d66-d4359e65bc86\") " pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:09.510368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.510332 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw92l\" (UniqueName: \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\") pod \"test-curl-8864-f40e45\" (UID: \"4ca5f0b9-6002-4a37-9d66-d4359e65bc86\") " pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:09.635298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.635169 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:09.943562 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:33:09.943535 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ca5f0b9_6002_4a37_9d66_d4359e65bc86.slice/crio-f97ad1bec825ed69ef34eaf622b261d19335758228b9b1e5556249e00fa5dca9 WatchSource:0}: Error finding container f97ad1bec825ed69ef34eaf622b261d19335758228b9b1e5556249e00fa5dca9: Status 404 returned error can't find the container with id f97ad1bec825ed69ef34eaf622b261d19335758228b9b1e5556249e00fa5dca9 Oct 30 05:33:09.946273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:09.946248 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8864-f40e45"] Oct 30 05:33:10.838428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:10.838391 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:33:10.838887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:10.838578 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" containerID="cri-o://7fe61d751c20aee10c653e6fccb9db6cac4715cfcb6ad25403cdafd1fbddcaa1" gracePeriod=30 Oct 30 05:33:10.851917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:10.851877 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:33:10.852130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:10.852107 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" containerID="cri-o://4be7209799743dbfadd969b62071c5c358f8111b3a50117f456d5087718a9842" gracePeriod=30 Oct 30 05:33:11.020240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.020001 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8864-f40e45" event={"ID":"4ca5f0b9-6002-4a37-9d66-d4359e65bc86","Type":"ContainerStarted","Data":"74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb"} Oct 30 05:33:11.020240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.020117 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8864-f40e45" event={"ID":"4ca5f0b9-6002-4a37-9d66-d4359e65bc86","Type":"ContainerStarted","Data":"f97ad1bec825ed69ef34eaf622b261d19335758228b9b1e5556249e00fa5dca9"} Oct 30 05:33:11.293547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.293468 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8864-f40e45" podStartSLOduration=2.293444187 podStartE2EDuration="2.293444187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:11.216450895 +0000 UTC m=+2964.185945881" watchObservedRunningTime="2026-10-30 05:33:11.293444187 +0000 UTC m=+2964.262939176" Oct 30 05:33:11.482102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.482067 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:33:11.482297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.482124 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:33:11.952181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.952144 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:33:11.968564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:11.968533 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 05:33:12.151208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:12.150965 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8864-f40e45"] Oct 30 05:33:12.158012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:12.157980 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8864-f40e45"] Oct 30 05:33:12.697511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:12.697475 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:33:12.697702 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:12.697550 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:33:13.036084 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:13.035997 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8864-f40e45" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" containerName="test-curl-8864-f40e45" containerID="cri-o://74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb" gracePeriod=2 Oct 30 05:33:13.051361 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:13.051326 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:15.279597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:15.279568 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:15.408193 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:15.408157 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\" (UniqueName: \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\") pod \"4ca5f0b9-6002-4a37-9d66-d4359e65bc86\" (UID: \"4ca5f0b9-6002-4a37-9d66-d4359e65bc86\") " Oct 30 05:33:15.410633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:15.410600 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l" pod "4ca5f0b9-6002-4a37-9d66-d4359e65bc86" (UID: "4ca5f0b9-6002-4a37-9d66-d4359e65bc86"). InnerVolumeSpecName "kube-api-access-tw92l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:15.510794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:15.510759 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw92l\" (UniqueName: \"kubernetes.io/projected/4ca5f0b9-6002-4a37-9d66-d4359e65bc86-kube-api-access-tw92l\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.067247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.066686 5058 generic.go:356] "Generic (PLEG): container finished" podID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" containerID="74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb" exitCode=137 Oct 30 05:33:16.067447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.066847 5058 scope.go:122] "RemoveContainer" containerID="74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb" Oct 30 05:33:16.067648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.066936 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:16.071884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.071844 5058 generic.go:356] "Generic (PLEG): container finished" podID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerID="7fe61d751c20aee10c653e6fccb9db6cac4715cfcb6ad25403cdafd1fbddcaa1" exitCode=0 Oct 30 05:33:16.072287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.072205 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" event={"ID":"d653fdb2-a81a-429f-838d-538ea3e59a11","Type":"ContainerDied","Data":"7fe61d751c20aee10c653e6fccb9db6cac4715cfcb6ad25403cdafd1fbddcaa1"} Oct 30 05:33:16.108944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.108758 5058 generic.go:356] "Generic (PLEG): container finished" podID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerID="4be7209799743dbfadd969b62071c5c358f8111b3a50117f456d5087718a9842" exitCode=0 Oct 30 05:33:16.109166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.108905 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" event={"ID":"6622fff6-ca8a-4ed5-869b-12f291249c2e","Type":"ContainerDied","Data":"4be7209799743dbfadd969b62071c5c358f8111b3a50117f456d5087718a9842"} Oct 30 05:33:16.116388 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.116364 5058 scope.go:122] "RemoveContainer" containerID="74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb" Oct 30 05:33:16.116890 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:16.116865 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb\": container with ID starting with 74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb not found: ID does not exist" containerID="74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb" Oct 30 05:33:16.117060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.117032 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb"} err="failed to get container status \"74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb\": rpc error: code = NotFound desc = could not find container \"74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb\": container with ID starting with 74c410e7750edee1ddcceda53b6bcc45d3c91d6ea99b42b0df43e4347ec01aeb not found: ID does not exist" Oct 30 05:33:16.129262 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:16.128010 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:16.247306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.247278 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:33:16.262489 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:16.262444 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:16.301851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.301824 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:33:16.342186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342142 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.342459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342271 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.342459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342304 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342334 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342412 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342460 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342533 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342557 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342585 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342622 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342686 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.342730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342728 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.343019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342780 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.343019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342805 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.343019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342905 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.343019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342962 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\") pod \"d653fdb2-a81a-429f-838d-538ea3e59a11\" (UID: \"d653fdb2-a81a-429f-838d-538ea3e59a11\") " Oct 30 05:33:16.343228 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.343037 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.343228 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.343076 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\") pod \"6622fff6-ca8a-4ed5-869b-12f291249c2e\" (UID: \"6622fff6-ca8a-4ed5-869b-12f291249c2e\") " Oct 30 05:33:16.346561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.342901 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.346561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.344271 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.346781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.344381 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.346781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.344609 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.346781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.345370 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.346781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.345448 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.346781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.346259 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.347059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.346856 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.348137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348109 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:16.348137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348126 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.348363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348177 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.348430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348393 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.348491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348446 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.348698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.348664 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.349146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.349093 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "kube-api-access-kkp5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.349337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.349311 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:16.350593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.350569 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy" pod "d653fdb2-a81a-429f-838d-538ea3e59a11" (UID: "d653fdb2-a81a-429f-838d-538ea3e59a11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.350887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.350863 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz" pod "6622fff6-ca8a-4ed5-869b-12f291249c2e" (UID: "6622fff6-ca8a-4ed5-869b-12f291249c2e"). InnerVolumeSpecName "kube-api-access-bfwjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.367473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.367373 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:33:16.368760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.368733 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" Oct 30 05:33:16.368923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.368907 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" Oct 30 05:33:16.369034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369021 5058 cpu_manager.go:414] "containerMap: removing container" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" Oct 30 05:33:16.369124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369113 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" Oct 30 05:33:16.369311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369296 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" containerName="test-curl-8864-f40e45" Oct 30 05:33:16.369423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369410 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" containerName="test-curl-8864-f40e45" Oct 30 05:33:16.369739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369721 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" containerName="istio-proxy" Oct 30 05:33:16.369863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369850 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" containerName="istio-proxy" Oct 30 05:33:16.369979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.369968 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" containerName="test-curl-8864-f40e45" Oct 30 05:33:16.376341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.376317 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.397644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.397613 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:33:16.420608 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:16.420564 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:16.435564 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:16.435520 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:16.445446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445414 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445480 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445660 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445686 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445768 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445892 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445936 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445960 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.445987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.445983 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mfc\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.446253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446230 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446260 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446274 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446288 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446301 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446315 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446327 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446341 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6622fff6-ca8a-4ed5-869b-12f291249c2e-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446350 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkp5g\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-kube-api-access-kkp5g\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446359 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446368 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwjz\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-kube-api-access-bfwjz\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446378 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446386 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6622fff6-ca8a-4ed5-869b-12f291249c2e-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446394 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446401 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d653fdb2-a81a-429f-838d-538ea3e59a11-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446410 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d653fdb2-a81a-429f-838d-538ea3e59a11-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446418 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d653fdb2-a81a-429f-838d-538ea3e59a11-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.446500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.446427 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6622fff6-ca8a-4ed5-869b-12f291249c2e-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.547458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547420 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547465 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547667 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547822 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547892 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547919 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.547971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547945 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mfc\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.547964 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548077 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548146 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548180 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548432 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548595 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.548961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.548932 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.550428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.550409 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.551071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.551044 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.570707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.570675 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.571657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.571633 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mfc\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:16.698346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:16.698299 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:17.034812 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:33:17.034778 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c5b1aa_0a5a_4663_bc8e_d2a815cdbc77.slice/crio-c669fb96229868160f7d00934c47c1323f2f9406a842cff1f6d27f02932d94a9 WatchSource:0}: Error finding container c669fb96229868160f7d00934c47c1323f2f9406a842cff1f6d27f02932d94a9: Status 404 returned error can't find the container with id c669fb96229868160f7d00934c47c1323f2f9406a842cff1f6d27f02932d94a9 Oct 30 05:33:17.037010 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.036983 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:33:17.038629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.038593 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:17.038730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.038675 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:17.038730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.038707 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:17.119150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.119122 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" Oct 30 05:33:17.119328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.119156 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n" event={"ID":"d653fdb2-a81a-429f-838d-538ea3e59a11","Type":"ContainerDied","Data":"e6dd5e686f05e0291b63f302ed412452e3bac73c197e66664e250427a59cbc01"} Oct 30 05:33:17.119328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.119274 5058 scope.go:122] "RemoveContainer" containerID="7fe61d751c20aee10c653e6fccb9db6cac4715cfcb6ad25403cdafd1fbddcaa1" Oct 30 05:33:17.121036 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.121014 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" Oct 30 05:33:17.121145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.121040 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt" event={"ID":"6622fff6-ca8a-4ed5-869b-12f291249c2e","Type":"ContainerDied","Data":"950074fb648674ff2264d58dc5cac0c8a399a8140aed640745a8d4215da83b6c"} Oct 30 05:33:17.128311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.125903 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" event={"ID":"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77","Type":"ContainerStarted","Data":"c669fb96229868160f7d00934c47c1323f2f9406a842cff1f6d27f02932d94a9"} Oct 30 05:33:17.135429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.134871 5058 scope.go:122] "RemoveContainer" containerID="4be7209799743dbfadd969b62071c5c358f8111b3a50117f456d5087718a9842" Oct 30 05:33:17.150833 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:17.150807 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:17.165659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.165627 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" path="/var/lib/kubelet/pods/4ca5f0b9-6002-4a37-9d66-d4359e65bc86/volumes" Oct 30 05:33:17.165659 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:17.165642 5058 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8864-f40e45\" is forbidden: User \"system:node:ip-10-0-139-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-228.ec2.internal' and this object" podUID="4ca5f0b9-6002-4a37-9d66-d4359e65bc86" pod="opendatahub/test-curl-8864-f40e45" Oct 30 05:33:17.218602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.218565 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:33:17.234747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.234715 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt"] Oct 30 05:33:17.328983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.328896 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:33:17.343069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:17.343030 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n"] Oct 30 05:33:18.212876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:18.212840 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" event={"ID":"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77","Type":"ContainerStarted","Data":"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9"} Oct 30 05:33:18.307660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:18.307604 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podStartSLOduration=181.307590577 podStartE2EDuration="3m1.307590577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:18.292511634 +0000 UTC m=+2971.262006622" watchObservedRunningTime="2026-10-30 05:33:18.307590577 +0000 UTC m=+2971.277085564" Oct 30 05:33:18.707344 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:18.707310 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:18.736312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:18.731471 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 05:33:18.736312 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:18.731574 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 05:33:19.165162 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.165126 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6622fff6-ca8a-4ed5-869b-12f291249c2e" path="/var/lib/kubelet/pods/6622fff6-ca8a-4ed5-869b-12f291249c2e/volumes" Oct 30 05:33:19.165749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.165728 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d653fdb2-a81a-429f-838d-538ea3e59a11" path="/var/lib/kubelet/pods/d653fdb2-a81a-429f-838d-538ea3e59a11/volumes" Oct 30 05:33:19.196239 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:19.196190 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:33:19.698983 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.698945 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 05:33:19.699184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.699004 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 05:33:19.756352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.756314 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:33:19.762609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.762588 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.786319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.786290 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:33:19.793984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.793957 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9dc17a-openshift-default-dockercfg-k442k\"" Oct 30 05:33:19.867812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.867776 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.867825 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.867956 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqfb\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868001 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868020 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868070 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868089 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868112 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.868384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.868168 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.969512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.969423 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.969767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.969747 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.969962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.969935 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.970111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970001 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqfb\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.970274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970259 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.970371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970360 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.970494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970484 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.970575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970553 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.971391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.970565 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.971605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.971586 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.971780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.971766 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.972262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.971011 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.972354 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.971330 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.972354 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.972081 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.979414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.974617 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:19.979414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:19.974984 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:20.007494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.001888 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqfb\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:20.007494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.002925 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:20.078171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.078139 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:20.454628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.454588 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:33:20.455935 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:33:20.455906 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785b3321_964e_48f9_804b_bb1b7c76b9dd.slice/crio-108a675daa53c630af69f2c7f0489a23e6c4b605487a98918637d57273716f64 WatchSource:0}: Error finding container 108a675daa53c630af69f2c7f0489a23e6c4b605487a98918637d57273716f64: Status 404 returned error can't find the container with id 108a675daa53c630af69f2c7f0489a23e6c4b605487a98918637d57273716f64 Oct 30 05:33:20.459025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.458993 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.459125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.459065 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.459125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.459094 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.699494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.699461 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 05:33:20.699667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:20.699514 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 05:33:21.286880 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.286843 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" event={"ID":"785b3321-964e-48f9-804b-bb1b7c76b9dd","Type":"ContainerStarted","Data":"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb"} Oct 30 05:33:21.287304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.286899 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" event={"ID":"785b3321-964e-48f9-804b-bb1b7c76b9dd","Type":"ContainerStarted","Data":"108a675daa53c630af69f2c7f0489a23e6c4b605487a98918637d57273716f64"} Oct 30 05:33:21.386293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.380169 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podStartSLOduration=2.380145678 podStartE2EDuration="2.380145678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:21.349881617 +0000 UTC m=+2974.319376606" watchObservedRunningTime="2026-10-30 05:33:21.380145678 +0000 UTC m=+2974.349640666" Oct 30 05:33:21.699355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.699320 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 05:33:21.699549 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.699371 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 05:33:21.991247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:21.988391 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:33:22.022409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.022366 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:33:22.096797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.088338 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:22.100093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.100057 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:22.100286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.100130 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:22.725310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.725266 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 05:33:22.738351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:22.738300 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 05:33:23.080944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:23.080841 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:23.080944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:23.080921 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:23.786740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:23.786686 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:23.787534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:23.787378 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:23.788484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:23.788460 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:33:24.085333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:24.085036 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:24.085333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:24.085105 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:25.078560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:25.078522 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:25.078997 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:25.078591 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:25.562245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:25.561928 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:33:25.562245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:25.562192 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" podUID="da39a40c-a60f-4b18-9107-eef438c08482" containerName="maas-api" containerID="cri-o://36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc" gracePeriod=30 Oct 30 05:33:26.210245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.085338 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:26.210875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.210840 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:26.482729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.482652 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:33:26.581451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.581415 5058 generic.go:356] "Generic (PLEG): container finished" podID="da39a40c-a60f-4b18-9107-eef438c08482" containerID="36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc" exitCode=0 Oct 30 05:33:26.581672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.581567 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" event={"ID":"da39a40c-a60f-4b18-9107-eef438c08482","Type":"ContainerDied","Data":"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc"} Oct 30 05:33:26.581672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.581610 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" event={"ID":"da39a40c-a60f-4b18-9107-eef438c08482","Type":"ContainerDied","Data":"cd874a52d20bc1606b7740d75d0893be3dc356f2163d1397832ee13b71df2a16"} Oct 30 05:33:26.581672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.581631 5058 scope.go:122] "RemoveContainer" containerID="36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc" Oct 30 05:33:26.581886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.581751 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws" Oct 30 05:33:26.599439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.599406 5058 scope.go:122] "RemoveContainer" containerID="36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc" Oct 30 05:33:26.599941 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:26.599907 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc\": container with ID starting with 36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc not found: ID does not exist" containerID="36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc" Oct 30 05:33:26.600054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.599954 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc"} err="failed to get container status \"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc\": rpc error: code = NotFound desc = could not find container \"36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc\": container with ID starting with 36f7232844914c21803ac5c5f0d05c35f06e5823e8a295e7a2b9b3f8e28e4bcc not found: ID does not exist" Oct 30 05:33:26.701305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.701267 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\" (UniqueName: \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\") pod \"da39a40c-a60f-4b18-9107-eef438c08482\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " Oct 30 05:33:26.701974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.701934 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\") pod \"da39a40c-a60f-4b18-9107-eef438c08482\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " Oct 30 05:33:26.710392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.710362 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\") pod \"da39a40c-a60f-4b18-9107-eef438c08482\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " Oct 30 05:33:26.710851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.710818 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\") pod \"da39a40c-a60f-4b18-9107-eef438c08482\" (UID: \"da39a40c-a60f-4b18-9107-eef438c08482\") " Oct 30 05:33:26.723394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.718398 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca" pod "da39a40c-a60f-4b18-9107-eef438c08482" (UID: "da39a40c-a60f-4b18-9107-eef438c08482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:26.723394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.719273 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls" pod "da39a40c-a60f-4b18-9107-eef438c08482" (UID: "da39a40c-a60f-4b18-9107-eef438c08482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:26.723394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.720230 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da39a40c-a60f-4b18-9107-eef438c08482-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:26.723394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.720253 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:26.746918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.746686 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56" pod "da39a40c-a60f-4b18-9107-eef438c08482" (UID: "da39a40c-a60f-4b18-9107-eef438c08482"). InnerVolumeSpecName "kube-api-access-r8n56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:26.761293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.754502 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs" pod "da39a40c-a60f-4b18-9107-eef438c08482" (UID: "da39a40c-a60f-4b18-9107-eef438c08482"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:26.821933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.821900 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da39a40c-a60f-4b18-9107-eef438c08482-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:26.821933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.821931 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8n56\" (UniqueName: \"kubernetes.io/projected/da39a40c-a60f-4b18-9107-eef438c08482-kube-api-access-r8n56\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:26.991242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.985913 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" containerID="cri-o://03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae" gracePeriod=45 Oct 30 05:33:27.001481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:26.999526 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" containerID="cri-o://fa472f1a95a8e46031d8d13a05957ca693636c22994b2d97ebcd5b9cd776c986" gracePeriod=25 Oct 30 05:33:27.008431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.008198 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" containerID="cri-o://4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015" gracePeriod=45 Oct 30 05:33:27.023745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.023691 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" containerID="cri-o://6818a61977e6849690bdb181a24321d9a974683251ab44f4c7a2fe5af5a9f8d3" gracePeriod=25 Oct 30 05:33:27.078181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.078128 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:33:27.078525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.078494 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 05:33:27.078635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.078553 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 05:33:27.083158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.083132 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2f6fed-d4d795d46-jfmws"] Oct 30 05:33:27.167093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.167049 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da39a40c-a60f-4b18-9107-eef438c08482" path="/var/lib/kubelet/pods/da39a40c-a60f-4b18-9107-eef438c08482/volumes" Oct 30 05:33:27.347250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.347202 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:33:27.372033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.371995 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:33:27.480274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.480201 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:33:27.481687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481659 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" Oct 30 05:33:27.481687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481685 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" Oct 30 05:33:27.481903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481713 5058 cpu_manager.go:414] "containerMap: removing container" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" Oct 30 05:33:27.481903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481721 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" Oct 30 05:33:27.481903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481745 5058 cpu_manager.go:414] "containerMap: removing container" podUID="da39a40c-a60f-4b18-9107-eef438c08482" containerName="maas-api" Oct 30 05:33:27.481903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.481753 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da39a40c-a60f-4b18-9107-eef438c08482" containerName="maas-api" Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482573 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482634 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482775 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482851 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\" (UniqueName: \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482987 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483047 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483080 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.483205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483116 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.484974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483790 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.484974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483909 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\" (UniqueName: \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\") pod \"fbc907e0-8871-4947-9711-43ead530bde5\" (UID: \"fbc907e0-8871-4947-9711-43ead530bde5\") " Oct 30 05:33:27.484974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.483984 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.484974 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.484055 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\") pod \"5a39c02e-7ca4-489a-b0cf-459359240c90\" (UID: \"5a39c02e-7ca4-489a-b0cf-459359240c90\") " Oct 30 05:33:27.486422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.486401 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.487252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.487229 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.487389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.487256 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.487869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.487564 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.487869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.482009 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerName="main" Oct 30 05:33:27.487869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.487761 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc907e0-8871-4947-9711-43ead530bde5" containerName="main" Oct 30 05:33:27.487869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.487781 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="da39a40c-a60f-4b18-9107-eef438c08482" containerName="maas-api" Oct 30 05:33:27.491603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.491580 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.491942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.491921 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.496821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.494902 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.505064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.505009 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.505660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.505631 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "kube-api-access-gg9jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:27.519968 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.518425 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs" pod "5a39c02e-7ca4-489a-b0cf-459359240c90" (UID: "5a39c02e-7ca4-489a-b0cf-459359240c90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:27.519968 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.518587 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "kube-api-access-tgshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:27.527588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.526719 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.527588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.527353 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs" pod "fbc907e0-8871-4947-9711-43ead530bde5" (UID: "fbc907e0-8871-4947-9711-43ead530bde5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:27.535245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.535035 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:33:27.562553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.562519 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4bc875-openshift-default-dockercfg-bdc5p\"" Oct 30 05:33:27.595204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.595108 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.603452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.603409 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.603714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.603692 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.635745 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.635705 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpbg\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.636048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.636019 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.636182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.636167 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.636410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.636384 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.653640 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.653597 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.673713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.673663 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.674212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.674185 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.754158 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.754132 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.754311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.754296 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.754568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.754537 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778528 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgshr\" (UniqueName: \"kubernetes.io/projected/fbc907e0-8871-4947-9711-43ead530bde5-kube-api-access-tgshr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778568 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778585 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a39c02e-7ca4-489a-b0cf-459359240c90-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778596 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778607 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a39c02e-7ca4-489a-b0cf-459359240c90-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778617 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc907e0-8871-4947-9711-43ead530bde5-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778628 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9jh\" (UniqueName: \"kubernetes.io/projected/5a39c02e-7ca4-489a-b0cf-459359240c90-kube-api-access-gg9jh\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.778643 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc907e0-8871-4947-9711-43ead530bde5-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.778817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.686299 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" Oct 30 05:33:27.779465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.686232 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" event={"ID":"5a39c02e-7ca4-489a-b0cf-459359240c90","Type":"ContainerDied","Data":"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae"} Oct 30 05:33:27.779577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.779508 5058 scope.go:122] "RemoveContainer" containerID="03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae" Oct 30 05:33:27.779677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.686178 5058 generic.go:356] "Generic (PLEG): container finished" podID="5a39c02e-7ca4-489a-b0cf-459359240c90" containerID="03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae" exitCode=0 Oct 30 05:33:27.779780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.779758 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl" event={"ID":"5a39c02e-7ca4-489a-b0cf-459359240c90","Type":"ContainerDied","Data":"2351597dfe9ace4ddb0ccfa2459d0e0cef8086f32d5b5bce25b9818a22446b07"} Oct 30 05:33:27.825916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.822779 5058 generic.go:356] "Generic (PLEG): container finished" podID="fbc907e0-8871-4947-9711-43ead530bde5" containerID="4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015" exitCode=0 Oct 30 05:33:27.825916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.822997 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" event={"ID":"fbc907e0-8871-4947-9711-43ead530bde5","Type":"ContainerDied","Data":"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015"} Oct 30 05:33:27.825916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.823049 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" event={"ID":"fbc907e0-8871-4947-9711-43ead530bde5","Type":"ContainerDied","Data":"20f08df842f10b03a40a1c97baa5ed7606a1e088c889a8547d40ebda47e33d75"} Oct 30 05:33:27.825916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.823135 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj" Oct 30 05:33:27.905242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.891274 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpbg\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.905653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.905611 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.905840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.905824 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.906016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.906003 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.906282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.906259 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.906499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.906476 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.906800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.906777 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.907070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.907053 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.907235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.907200 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.909688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.909664 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.910908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.910875 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.931502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.915533 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.931502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.923493 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.931502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.923736 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.931502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.923972 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.932248 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.931787 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.936835 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.936769 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.938414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.937799 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpbg\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\") pod \"e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:27.989558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.989531 5058 scope.go:122] "RemoveContainer" containerID="03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae" Oct 30 05:33:27.990004 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:27.989973 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae\": container with ID starting with 03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae not found: ID does not exist" containerID="03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae" Oct 30 05:33:27.990090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.990016 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae"} err="failed to get container status \"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae\": rpc error: code = NotFound desc = could not find container \"03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae\": container with ID starting with 03700ad62d5afb135a92686b5cc95a5845ef1dd21f1fc03993f9a8fc39c2d0ae not found: ID does not exist" Oct 30 05:33:27.990090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:27.990042 5058 scope.go:122] "RemoveContainer" containerID="4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015" Oct 30 05:33:28.022694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.022667 5058 scope.go:122] "RemoveContainer" containerID="4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015" Oct 30 05:33:28.023041 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:28.023009 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015\": container with ID starting with 4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015 not found: ID does not exist" containerID="4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015" Oct 30 05:33:28.023127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.023056 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015"} err="failed to get container status \"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015\": rpc error: code = NotFound desc = could not find container \"4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015\": container with ID starting with 4eae766f14de8c0f37779ef7cf9829197f1f2c6bdd17a753329cfe9684887015 not found: ID does not exist" Oct 30 05:33:28.049982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.049949 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:33:28.060680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.060642 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-2-w3-889ca9-kserve-bd9fb8667-q48nl"] Oct 30 05:33:28.104054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.104016 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 05:33:28.115708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.115676 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/distinct-w3-889ca9-kserve-7546d87c76-q4jrj"] Oct 30 05:33:28.127568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.127534 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:28.128694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.128668 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:28.129049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.129029 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:33:28.147986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.138549 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:28.239711 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.239678 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:28.240163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.240138 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:28.512414 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:33:28.512385 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43bbdb4_3917_460e_8c18_d81926232c43.slice/crio-0b73110aee0c1c16c78f45072d6ac83b30e72a01061bacea632ec5ad841f74af WatchSource:0}: Error finding container 0b73110aee0c1c16c78f45072d6ac83b30e72a01061bacea632ec5ad841f74af: Status 404 returned error can't find the container with id 0b73110aee0c1c16c78f45072d6ac83b30e72a01061bacea632ec5ad841f74af Oct 30 05:33:28.514204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.514177 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:33:28.515143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.515113 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:28.515205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.515178 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:28.515263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.515209 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:28.834147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.834042 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerStarted","Data":"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f"} Oct 30 05:33:28.834147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.834112 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerStarted","Data":"0b73110aee0c1c16c78f45072d6ac83b30e72a01061bacea632ec5ad841f74af"} Oct 30 05:33:28.889000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:28.888932 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podStartSLOduration=1.888911624 podStartE2EDuration="1.888911624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:28.882704035 +0000 UTC m=+2981.852199023" watchObservedRunningTime="2026-10-30 05:33:28.888911624 +0000 UTC m=+2981.858406613" Oct 30 05:33:29.142477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:29.142432 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:29.144433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:29.144404 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:33:29.144567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:29.144453 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:33:29.165366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:29.165333 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a39c02e-7ca4-489a-b0cf-459359240c90" path="/var/lib/kubelet/pods/5a39c02e-7ca4-489a-b0cf-459359240c90/volumes" Oct 30 05:33:29.165923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:29.165905 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc907e0-8871-4947-9711-43ead530bde5" path="/var/lib/kubelet/pods/fbc907e0-8871-4947-9711-43ead530bde5/volumes" Oct 30 05:33:30.146674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:30.146639 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:30.853297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:30.853263 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:30.854210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:30.854190 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:33:34.900619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:34.894525 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:33:34.900619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:34.894824 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" containerID="cri-o://3ec2f29dbbdc0fca0f0976c0fd4699c68ff35b92a5c8a35e9e9c0178b4aefe0e" gracePeriod=30 Oct 30 05:33:37.188871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:37.188826 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:33:38.239201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:38.239164 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:38.239645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:38.239534 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:40.149743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.149702 5058 generic.go:356] "Generic (PLEG): container finished" podID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerID="3ec2f29dbbdc0fca0f0976c0fd4699c68ff35b92a5c8a35e9e9c0178b4aefe0e" exitCode=0 Oct 30 05:33:40.150211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.149769 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerDied","Data":"3ec2f29dbbdc0fca0f0976c0fd4699c68ff35b92a5c8a35e9e9c0178b4aefe0e"} Oct 30 05:33:40.150211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.149829 5058 scope.go:122] "RemoveContainer" containerID="987fd5b39a006fc3bec357e67441f619d2b512064584ddf81dce0d939bf9e39e" Oct 30 05:33:40.205736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.205678 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:33:40.329664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329623 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.329858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329734 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.329858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329809 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.329858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329843 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329949 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330004 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330128 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330154 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330156 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:40.330289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330172 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:40.330617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330294 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\") pod \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\" (UID: \"bf1bbf5b-7760-4207-b379-b4ee036f8eb8\") " Oct 30 05:33:40.330617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330455 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:40.330742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.330704 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:40.330870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.329866 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:40.331180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.331100 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.331180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.331129 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.331180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.331145 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.331180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.331159 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.331180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.331173 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.333305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.333274 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:40.333415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.333330 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:40.338603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.338575 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:40.338694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.338650 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77" pod "bf1bbf5b-7760-4207-b379-b4ee036f8eb8" (UID: "bf1bbf5b-7760-4207-b379-b4ee036f8eb8"). InnerVolumeSpecName "kube-api-access-s8z77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:40.432587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.432549 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.432587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.432582 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.432587 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.432596 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.432837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:40.432608 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8z77\" (UniqueName: \"kubernetes.io/projected/bf1bbf5b-7760-4207-b379-b4ee036f8eb8-kube-api-access-s8z77\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:41.161064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:41.161035 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" Oct 30 05:33:41.164445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:41.164414 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt" event={"ID":"bf1bbf5b-7760-4207-b379-b4ee036f8eb8","Type":"ContainerDied","Data":"8205346d6f712c6e9e82c2d012c3841797718d8638c7456da69846a5d7b59498"} Oct 30 05:33:41.164628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:41.164470 5058 scope.go:122] "RemoveContainer" containerID="3ec2f29dbbdc0fca0f0976c0fd4699c68ff35b92a5c8a35e9e9c0178b4aefe0e" Oct 30 05:33:41.238165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:41.238124 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:33:41.251343 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:41.251312 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt"] Oct 30 05:33:43.165722 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:43.165688 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" path="/var/lib/kubelet/pods/bf1bbf5b-7760-4207-b379-b4ee036f8eb8/volumes" Oct 30 05:33:47.295764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.295728 5058 generic.go:356] "Generic (PLEG): container finished" podID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerID="fa472f1a95a8e46031d8d13a05957ca693636c22994b2d97ebcd5b9cd776c986" exitCode=0 Oct 30 05:33:47.296285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.295880 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" event={"ID":"4c7c7593-101f-4678-ad91-dad68d4bb26b","Type":"ContainerDied","Data":"fa472f1a95a8e46031d8d13a05957ca693636c22994b2d97ebcd5b9cd776c986"} Oct 30 05:33:47.297602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.297572 5058 generic.go:356] "Generic (PLEG): container finished" podID="59448e72-ab39-48b0-b703-66ab864916b3" containerID="6818a61977e6849690bdb181a24321d9a974683251ab44f4c7a2fe5af5a9f8d3" exitCode=0 Oct 30 05:33:47.297786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.297769 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" event={"ID":"59448e72-ab39-48b0-b703-66ab864916b3","Type":"ContainerDied","Data":"6818a61977e6849690bdb181a24321d9a974683251ab44f4c7a2fe5af5a9f8d3"} Oct 30 05:33:47.524207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.524166 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:33:47.662464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.662433 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:33:47.669416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.669376 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\" (UniqueName: \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\") pod \"4c7c7593-101f-4678-ad91-dad68d4bb26b\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " Oct 30 05:33:47.669620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.669569 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\") pod \"4c7c7593-101f-4678-ad91-dad68d4bb26b\" (UID: \"4c7c7593-101f-4678-ad91-dad68d4bb26b\") " Oct 30 05:33:47.670038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.670008 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume" pod "4c7c7593-101f-4678-ad91-dad68d4bb26b" (UID: "4c7c7593-101f-4678-ad91-dad68d4bb26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:47.670591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.670564 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c7593-101f-4678-ad91-dad68d4bb26b-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:47.672358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.672332 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2" pod "4c7c7593-101f-4678-ad91-dad68d4bb26b" (UID: "4c7c7593-101f-4678-ad91-dad68d4bb26b"). InnerVolumeSpecName "kube-api-access-rpsz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:47.772167 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.772128 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\") pod \"59448e72-ab39-48b0-b703-66ab864916b3\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " Oct 30 05:33:47.772542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.772521 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\" (UniqueName: \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\") pod \"59448e72-ab39-48b0-b703-66ab864916b3\" (UID: \"59448e72-ab39-48b0-b703-66ab864916b3\") " Oct 30 05:33:47.773511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.773488 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsz2\" (UniqueName: \"kubernetes.io/projected/4c7c7593-101f-4678-ad91-dad68d4bb26b-kube-api-access-rpsz2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:47.773674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.773646 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume" pod "59448e72-ab39-48b0-b703-66ab864916b3" (UID: "59448e72-ab39-48b0-b703-66ab864916b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:47.776641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.776608 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd" pod "59448e72-ab39-48b0-b703-66ab864916b3" (UID: "59448e72-ab39-48b0-b703-66ab864916b3"). InnerVolumeSpecName "kube-api-access-xrpvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:47.817079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.817057 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:33:47.821899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.821876 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:33:47.874965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.874871 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59448e72-ab39-48b0-b703-66ab864916b3-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:47.874965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:47.874909 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpvd\" (UniqueName: \"kubernetes.io/projected/59448e72-ab39-48b0-b703-66ab864916b3-kube-api-access-xrpvd\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.307955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.307734 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" Oct 30 05:33:48.307955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.307768 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7" event={"ID":"4c7c7593-101f-4678-ad91-dad68d4bb26b","Type":"ContainerDied","Data":"82a2791e3501bf193fa9883aee22381d54ef7e072206279063fad50a806bb5fb"} Oct 30 05:33:48.307955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.307829 5058 scope.go:122] "RemoveContainer" containerID="fa472f1a95a8e46031d8d13a05957ca693636c22994b2d97ebcd5b9cd776c986" Oct 30 05:33:48.309424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.309398 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" event={"ID":"59448e72-ab39-48b0-b703-66ab864916b3","Type":"ContainerDied","Data":"5a24cd6f115bbaf893d31f74dbf787193c72886f04f3003c41ddc532c92435bd"} Oct 30 05:33:48.309510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.309477 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2" Oct 30 05:33:48.321434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.321413 5058 scope.go:122] "RemoveContainer" containerID="6818a61977e6849690bdb181a24321d9a974683251ab44f4c7a2fe5af5a9f8d3" Oct 30 05:33:48.382507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.382473 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:33:48.395945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.395914 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2"] Oct 30 05:33:48.450314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.449765 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:33:48.457462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:48.457432 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7"] Oct 30 05:33:49.163699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:49.163664 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" path="/var/lib/kubelet/pods/4c7c7593-101f-4678-ad91-dad68d4bb26b/volumes" Oct 30 05:33:49.164271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:49.164252 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59448e72-ab39-48b0-b703-66ab864916b3" path="/var/lib/kubelet/pods/59448e72-ab39-48b0-b703-66ab864916b3/volumes" Oct 30 05:33:49.195507 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:33:49.195471 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:33:52.213419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.213366 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" podUID="58ef2fa7-5a82-4019-89b3-39595004d62c" containerName="main" containerID="cri-o://3423212e92be73794f14770a8d90b916f9df19aa0acb359959b9b72e303fafd3" gracePeriod=45 Oct 30 05:33:52.404967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.403676 5058 generic.go:356] "Generic (PLEG): container finished" podID="58ef2fa7-5a82-4019-89b3-39595004d62c" containerID="3423212e92be73794f14770a8d90b916f9df19aa0acb359959b9b72e303fafd3" exitCode=0 Oct 30 05:33:52.404967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.403912 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" event={"ID":"58ef2fa7-5a82-4019-89b3-39595004d62c","Type":"ContainerDied","Data":"3423212e92be73794f14770a8d90b916f9df19aa0acb359959b9b72e303fafd3"} Oct 30 05:33:52.548915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.548881 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:33:52.581389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581345 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\" (UniqueName: \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581441 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\" (UniqueName: \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581486 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581562 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581646 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581681 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\") pod \"58ef2fa7-5a82-4019-89b3-39595004d62c\" (UID: \"58ef2fa7-5a82-4019-89b3-39595004d62c\") " Oct 30 05:33:52.581972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581856 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:52.581972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.581957 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:52.582322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.582293 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:52.582492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.582468 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:52.582564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.582498 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:52.582564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.582511 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:52.583914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.583889 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:52.584018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.583994 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "kube-api-access-qfw8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:52.584095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.584044 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm" pod "58ef2fa7-5a82-4019-89b3-39595004d62c" (UID: "58ef2fa7-5a82-4019-89b3-39595004d62c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:52.684441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.684401 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ef2fa7-5a82-4019-89b3-39595004d62c-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:52.684441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.684442 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ef2fa7-5a82-4019-89b3-39595004d62c-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:52.684655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:52.684459 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfw8b\" (UniqueName: \"kubernetes.io/projected/58ef2fa7-5a82-4019-89b3-39595004d62c-kube-api-access-qfw8b\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:33:53.415699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:53.415671 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" Oct 30 05:33:53.415699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:53.415677 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx" event={"ID":"58ef2fa7-5a82-4019-89b3-39595004d62c","Type":"ContainerDied","Data":"526a44162a2a294214fe038b21500dee1ae30a3ce02de58e0a4139f204e03cae"} Oct 30 05:33:53.416208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:53.415744 5058 scope.go:122] "RemoveContainer" containerID="3423212e92be73794f14770a8d90b916f9df19aa0acb359959b9b72e303fafd3" Oct 30 05:33:53.474376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:53.474324 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:33:53.487677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:53.487646 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9dc17a/auth-test-model-9dc17a-kserve-6c877997c7-rxdpx"] Oct 30 05:33:54.425855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.425819 5058 generic.go:356] "Generic (PLEG): container finished" podID="2340d19f-7876-4594-8113-7ae42c13e555" containerID="591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617" exitCode=0 Oct 30 05:33:54.426409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.425887 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerDied","Data":"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617"} Oct 30 05:33:54.426505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.426458 5058 scope.go:122] "RemoveContainer" containerID="591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617" Oct 30 05:33:54.428562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.428530 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:54.428670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.428616 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:54.428744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:54.428670 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:55.178280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:55.178198 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ef2fa7-5a82-4019-89b3-39595004d62c" path="/var/lib/kubelet/pods/58ef2fa7-5a82-4019-89b3-39595004d62c/volumes" Oct 30 05:33:55.438083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:55.437996 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerStarted","Data":"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189"} Oct 30 05:33:56.124200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:56.124163 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:33:56.132242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:56.130350 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:33:56.132242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:56.130435 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:33:56.864324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:56.864286 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:33:57.124320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:57.124204 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:33:57.124320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:57.124294 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:33:58.143303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:58.135345 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:33:58.143303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:58.135426 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:33:59.135243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:59.134609 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:33:59.135243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:33:59.134671 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:34:00.136249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:00.134610 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:34:00.136249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:00.134674 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:34:01.124187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:01.124147 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 05:34:01.124425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:01.124302 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 05:34:01.866242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:01.865954 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" podUID="c08c6510-b69a-40f3-beb5-8b4428c8e695" containerName="payload-processing" containerID="cri-o://735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3" gracePeriod=25 Oct 30 05:34:01.946837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:01.945607 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:34:02.223957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.223923 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:34:02.274203 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.274166 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:34:02.294140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.294106 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\") pod \"c08c6510-b69a-40f3-beb5-8b4428c8e695\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " Oct 30 05:34:02.294356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.294290 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\" (UniqueName: \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\") pod \"c08c6510-b69a-40f3-beb5-8b4428c8e695\" (UID: \"c08c6510-b69a-40f3-beb5-8b4428c8e695\") " Oct 30 05:34:02.294551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.294528 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume" pod "c08c6510-b69a-40f3-beb5-8b4428c8e695" (UID: "c08c6510-b69a-40f3-beb5-8b4428c8e695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:02.295021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.294992 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08c6510-b69a-40f3-beb5-8b4428c8e695-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:02.296613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.296587 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p" pod "c08c6510-b69a-40f3-beb5-8b4428c8e695" (UID: "c08c6510-b69a-40f3-beb5-8b4428c8e695"). InnerVolumeSpecName "kube-api-access-z5f9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:02.396148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.396115 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5f9p\" (UniqueName: \"kubernetes.io/projected/c08c6510-b69a-40f3-beb5-8b4428c8e695-kube-api-access-z5f9p\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:02.510625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.510548 5058 generic.go:356] "Generic (PLEG): container finished" podID="c08c6510-b69a-40f3-beb5-8b4428c8e695" containerID="735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3" exitCode=0 Oct 30 05:34:02.510625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.510589 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" Oct 30 05:34:02.510831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.510636 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" event={"ID":"c08c6510-b69a-40f3-beb5-8b4428c8e695","Type":"ContainerDied","Data":"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3"} Oct 30 05:34:02.510831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.510692 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8" event={"ID":"c08c6510-b69a-40f3-beb5-8b4428c8e695","Type":"ContainerDied","Data":"1a06911dcb419aa11be8add7d7841899432e05a742dd9a1e6520594dccb2cbc5"} Oct 30 05:34:02.510831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.510721 5058 scope.go:122] "RemoveContainer" containerID="735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3" Oct 30 05:34:02.512415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.512391 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:34:02.523422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.523404 5058 scope.go:122] "RemoveContainer" containerID="735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3" Oct 30 05:34:02.523695 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:02.523679 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3\": container with ID starting with 735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3 not found: ID does not exist" containerID="735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3" Oct 30 05:34:02.523741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.523704 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3"} err="failed to get container status \"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3\": rpc error: code = NotFound desc = could not find container \"735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3\": container with ID starting with 735f50f47cc442d8c226abe872b9f48f07ed95717155084ce29d532009dcc9e3 not found: ID does not exist" Oct 30 05:34:02.568963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.568924 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:34:02.578372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:02.578343 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8"] Oct 30 05:34:03.164536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:03.164503 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08c6510-b69a-40f3-beb5-8b4428c8e695" path="/var/lib/kubelet/pods/c08c6510-b69a-40f3-beb5-8b4428c8e695/volumes" Oct 30 05:34:10.477079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:10.477043 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:34:10.510875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:10.510838 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:34:15.478301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.478241 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" podUID="68f2817e-ca70-4487-b33f-6274df124564" containerName="payload-processing" containerID="cri-o://798f8d6ed605fdada4c9a1de795cb8476792ec10781e6d084333a1bbeacb98ed" gracePeriod=25 Oct 30 05:34:15.511296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.511205 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" podUID="c3a2571c-152e-442d-bb3b-a52ebf896e65" containerName="payload-pre-processing" containerID="cri-o://02495b88fd3d105e4cf56fe5760c1560045317486365d32314c62fa721ca11f3" gracePeriod=25 Oct 30 05:34:15.636164 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.636132 5058 generic.go:356] "Generic (PLEG): container finished" podID="68f2817e-ca70-4487-b33f-6274df124564" containerID="798f8d6ed605fdada4c9a1de795cb8476792ec10781e6d084333a1bbeacb98ed" exitCode=0 Oct 30 05:34:15.636370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.636211 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" event={"ID":"68f2817e-ca70-4487-b33f-6274df124564","Type":"ContainerDied","Data":"798f8d6ed605fdada4c9a1de795cb8476792ec10781e6d084333a1bbeacb98ed"} Oct 30 05:34:15.638588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.638552 5058 generic.go:356] "Generic (PLEG): container finished" podID="c3a2571c-152e-442d-bb3b-a52ebf896e65" containerID="02495b88fd3d105e4cf56fe5760c1560045317486365d32314c62fa721ca11f3" exitCode=0 Oct 30 05:34:15.638724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.638621 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" event={"ID":"c3a2571c-152e-442d-bb3b-a52ebf896e65","Type":"ContainerDied","Data":"02495b88fd3d105e4cf56fe5760c1560045317486365d32314c62fa721ca11f3"} Oct 30 05:34:15.847766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.847736 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:34:15.865005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.864976 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\" (UniqueName: \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\") pod \"68f2817e-ca70-4487-b33f-6274df124564\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " Oct 30 05:34:15.865205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.865171 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\") pod \"68f2817e-ca70-4487-b33f-6274df124564\" (UID: \"68f2817e-ca70-4487-b33f-6274df124564\") " Oct 30 05:34:15.865661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.865550 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume" pod "68f2817e-ca70-4487-b33f-6274df124564" (UID: "68f2817e-ca70-4487-b33f-6274df124564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:15.865887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.865869 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2817e-ca70-4487-b33f-6274df124564-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:15.867513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.867488 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st" pod "68f2817e-ca70-4487-b33f-6274df124564" (UID: "68f2817e-ca70-4487-b33f-6274df124564"). InnerVolumeSpecName "kube-api-access-v44st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:15.899504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.899475 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:34:15.966510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.966478 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\" (UniqueName: \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\") pod \"c3a2571c-152e-442d-bb3b-a52ebf896e65\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " Oct 30 05:34:15.966860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.966839 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\") pod \"c3a2571c-152e-442d-bb3b-a52ebf896e65\" (UID: \"c3a2571c-152e-442d-bb3b-a52ebf896e65\") " Oct 30 05:34:15.967295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.967263 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume" pod "c3a2571c-152e-442d-bb3b-a52ebf896e65" (UID: "c3a2571c-152e-442d-bb3b-a52ebf896e65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:15.967560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.967537 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44st\" (UniqueName: \"kubernetes.io/projected/68f2817e-ca70-4487-b33f-6274df124564-kube-api-access-v44st\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:15.967639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.967564 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a2571c-152e-442d-bb3b-a52ebf896e65-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:15.968787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:15.968763 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf" pod "c3a2571c-152e-442d-bb3b-a52ebf896e65" (UID: "c3a2571c-152e-442d-bb3b-a52ebf896e65"). InnerVolumeSpecName "kube-api-access-86nzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:16.069449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.069365 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nzf\" (UniqueName: \"kubernetes.io/projected/c3a2571c-152e-442d-bb3b-a52ebf896e65-kube-api-access-86nzf\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:16.648357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.648324 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" Oct 30 05:34:16.648357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.648337 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm" event={"ID":"68f2817e-ca70-4487-b33f-6274df124564","Type":"ContainerDied","Data":"380909cce646ffacfd131d0aa6cca3b677e492eeaf4b67391739fdb27c78bb57"} Oct 30 05:34:16.648940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.648402 5058 scope.go:122] "RemoveContainer" containerID="798f8d6ed605fdada4c9a1de795cb8476792ec10781e6d084333a1bbeacb98ed" Oct 30 05:34:16.650019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.649994 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" Oct 30 05:34:16.650126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.650032 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w" event={"ID":"c3a2571c-152e-442d-bb3b-a52ebf896e65","Type":"ContainerDied","Data":"f2eb4ad5aeb8f785e8053ca2e16b96cb168dd78aa75593d5c65183d956da3d14"} Oct 30 05:34:16.662411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.662383 5058 scope.go:122] "RemoveContainer" containerID="02495b88fd3d105e4cf56fe5760c1560045317486365d32314c62fa721ca11f3" Oct 30 05:34:16.708836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.708796 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:34:16.722094 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.722060 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w"] Oct 30 05:34:16.760944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.760861 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:34:16.767744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:16.767712 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm"] Oct 30 05:34:17.163845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:17.163808 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f2817e-ca70-4487-b33f-6274df124564" path="/var/lib/kubelet/pods/68f2817e-ca70-4487-b33f-6274df124564/volumes" Oct 30 05:34:17.164372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:17.164356 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a2571c-152e-442d-bb3b-a52ebf896e65" path="/var/lib/kubelet/pods/c3a2571c-152e-442d-bb3b-a52ebf896e65/volumes" Oct 30 05:34:19.192208 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:19.192162 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:34:20.150770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.150735 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:34:20.151631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151607 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c08c6510-b69a-40f3-beb5-8b4428c8e695" containerName="payload-processing" Oct 30 05:34:20.151631 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151629 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08c6510-b69a-40f3-beb5-8b4428c8e695" containerName="payload-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151670 5058 cpu_manager.go:414] "containerMap: removing container" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151676 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151693 5058 cpu_manager.go:414] "containerMap: removing container" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151698 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151707 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151712 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151719 5058 cpu_manager.go:414] "containerMap: removing container" podUID="68f2817e-ca70-4487-b33f-6274df124564" containerName="payload-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151723 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f2817e-ca70-4487-b33f-6274df124564" containerName="payload-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151739 5058 cpu_manager.go:414] "containerMap: removing container" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151745 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151755 5058 cpu_manager.go:414] "containerMap: removing container" podUID="58ef2fa7-5a82-4019-89b3-39595004d62c" containerName="main" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151760 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ef2fa7-5a82-4019-89b3-39595004d62c" containerName="main" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151774 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c3a2571c-152e-442d-bb3b-a52ebf896e65" containerName="payload-pre-processing" Oct 30 05:34:20.151824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151779 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a2571c-152e-442d-bb3b-a52ebf896e65" containerName="payload-pre-processing" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151893 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="59448e72-ab39-48b0-b703-66ab864916b3" containerName="payload-pre-processing" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151905 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151912 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7c7593-101f-4678-ad91-dad68d4bb26b" containerName="payload-processing" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151921 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f2817e-ca70-4487-b33f-6274df124564" containerName="payload-processing" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151934 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1bbf5b-7760-4207-b379-b4ee036f8eb8" containerName="istio-proxy" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151942 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ef2fa7-5a82-4019-89b3-39595004d62c" containerName="main" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151948 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a2571c-152e-442d-bb3b-a52ebf896e65" containerName="payload-pre-processing" Oct 30 05:34:20.152367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.151955 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08c6510-b69a-40f3-beb5-8b4428c8e695" containerName="payload-processing" Oct 30 05:34:20.156494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.156477 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.186534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.184909 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4bc875-serving-cert\"" Oct 30 05:34:20.193270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.191824 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:34:20.258126 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.258087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljbw\" (UniqueName: \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.258504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.258476 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.258604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.258569 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.258670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.258621 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.360096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.360049 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljbw\" (UniqueName: \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.360374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.360353 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.360480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.360449 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.360540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.360504 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.361341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.361313 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.363200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.363176 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.363351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.363333 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.384859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.384823 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljbw\" (UniqueName: \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\") pod \"maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.484308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.483172 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:20.855140 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:34:20.855102 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1e3ee8_1dc8_410b_8eb9_e7a15a6888d6.slice/crio-e7a87fe7ff266a6f9efd12cbd9e5887241b3b6b043f90da31164c868f0231c11 WatchSource:0}: Error finding container e7a87fe7ff266a6f9efd12cbd9e5887241b3b6b043f90da31164c868f0231c11: Status 404 returned error can't find the container with id e7a87fe7ff266a6f9efd12cbd9e5887241b3b6b043f90da31164c868f0231c11 Oct 30 05:34:20.856310 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.856283 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:34:20.856620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:20.856603 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:34:21.773633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:21.772108 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" event={"ID":"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6","Type":"ContainerStarted","Data":"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49"} Oct 30 05:34:21.773633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:21.772175 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" event={"ID":"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6","Type":"ContainerStarted","Data":"e7a87fe7ff266a6f9efd12cbd9e5887241b3b6b043f90da31164c868f0231c11"} Oct 30 05:34:21.773633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:21.772466 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:21.822189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:21.822129 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" podStartSLOduration=1.3947505470000001 podStartE2EDuration="1.822112079s" totalImagesPullingTime="427.361532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:21.817290725 +0000 UTC m=+3034.786785719" watchObservedRunningTime="2026-10-30 05:34:21.822112079 +0000 UTC m=+3034.791607068" Oct 30 05:34:27.786765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:27.786735 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:34:29.311402 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.311361 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:34:29.330266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.318615 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.367913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.367879 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.368323 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.368300 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.368525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.368506 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.368690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.368675 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pxp\" (UniqueName: \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.372795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.372758 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:34:29.393303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.390369 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7226a8-serving-cert\"" Oct 30 05:34:29.472317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.470861 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.472317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.470996 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.472317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.471056 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.472317 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.471118 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pxp\" (UniqueName: \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.473789 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.473740 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.473976 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:29.473911 5058 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-serving-cert: secret "maas-api-e2e-ipp-a-7226a8-serving-cert" not found Oct 30 05:34:29.474152 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:29.474139 5058 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls podName:adb747f5-5af4-4af4-b359-92de31f91784 nodeName:}" failed. No retries permitted until 2026-10-30 05:34:29.974115844 +0000 UTC m=+3042.943610825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls") pod "maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" (UID: "adb747f5-5af4-4af4-b359-92de31f91784") : secret "maas-api-e2e-ipp-a-7226a8-serving-cert" not found Oct 30 05:34:29.475008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.474958 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:29.498713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:29.498676 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pxp\" (UniqueName: \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:30.010450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.010409 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:30.012990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.012957 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:30.013269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.013209 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:30.331367 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:34:30.331333 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb747f5_5af4_4af4_b359_92de31f91784.slice/crio-4d7b663276cbee775b94a3b29ebc2f83904f4425ebb8bff8c1b7d04b081c3d4e WatchSource:0}: Error finding container 4d7b663276cbee775b94a3b29ebc2f83904f4425ebb8bff8c1b7d04b081c3d4e: Status 404 returned error can't find the container with id 4d7b663276cbee775b94a3b29ebc2f83904f4425ebb8bff8c1b7d04b081c3d4e Oct 30 05:34:30.331954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.331925 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:34:30.942298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.941342 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" event={"ID":"adb747f5-5af4-4af4-b359-92de31f91784","Type":"ContainerStarted","Data":"9bb4b7e171faec4ee0ac3d0acc67aac45d682445ca0f0d6146203b1b95d80c1c"} Oct 30 05:34:30.942298 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.941433 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" event={"ID":"adb747f5-5af4-4af4-b359-92de31f91784","Type":"ContainerStarted","Data":"4d7b663276cbee775b94a3b29ebc2f83904f4425ebb8bff8c1b7d04b081c3d4e"} Oct 30 05:34:30.947246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:30.942800 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:31.004247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:31.003420 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" podStartSLOduration=1.644803099 podStartE2EDuration="2.003399016s" totalImagesPullingTime="358.595917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:31.002742896 +0000 UTC m=+3043.972237890" watchObservedRunningTime="2026-10-30 05:34:31.003399016 +0000 UTC m=+3043.972894006" Oct 30 05:34:37.563713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:37.563666 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:34:37.564257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:37.563927 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" containerID="cri-o://08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96" gracePeriod=30 Oct 30 05:34:38.029682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:38.029645 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:34:38.959808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:38.959762 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:34:39.581909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:39.581870 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:34:39.582096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:39.581952 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:34:42.879778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.879749 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:34:42.980396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980348 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980455 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980641 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980729 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980778 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980813 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.980915 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980899 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.981131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.980966 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.981131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.981007 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\") pod \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\" (UID: \"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049\") " Oct 30 05:34:42.986458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.986418 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:42.987281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.987196 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:42.988265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.988201 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:42.988428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.988400 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:42.988692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.988471 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:42.990143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.990113 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:42.991351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.991318 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "kube-api-access-7mrqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:42.993391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:42.991317 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:34:43.000454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.000418 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token" pod "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" (UID: "0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:43.086341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086296 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086337 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086350 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086363 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086377 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086392 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086406 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrqr\" (UniqueName: \"kubernetes.io/projected/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-kube-api-access-7mrqr\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086419 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.086682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.086433 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.212841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.212799 5058 generic.go:356] "Generic (PLEG): container finished" podID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerID="08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96" exitCode=0 Oct 30 05:34:43.213063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.212889 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" Oct 30 05:34:43.213063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.212891 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" event={"ID":"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049","Type":"ContainerDied","Data":"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96"} Oct 30 05:34:43.213063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.213004 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6" event={"ID":"0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049","Type":"ContainerDied","Data":"2d99daa67a35319382b8ac7a9c643373d854ecdcd4f6a720a4f3ac93c4b09782"} Oct 30 05:34:43.213063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.213039 5058 scope.go:122] "RemoveContainer" containerID="08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96" Oct 30 05:34:43.224588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.224562 5058 scope.go:122] "RemoveContainer" containerID="08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96" Oct 30 05:34:43.224911 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:43.224884 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96\": container with ID starting with 08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96 not found: ID does not exist" containerID="08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96" Oct 30 05:34:43.225006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.224921 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96"} err="failed to get container status \"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96\": rpc error: code = NotFound desc = could not find container \"08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96\": container with ID starting with 08e75034dea9639b90feced59d0d0881ec37b0a549567ad921743085423a1c96 not found: ID does not exist" Oct 30 05:34:43.272656 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.272623 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:34:43.278412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:43.278379 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6"] Oct 30 05:34:45.224790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.224747 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" path="/var/lib/kubelet/pods/0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049/volumes" Oct 30 05:34:45.399363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.399304 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:34:45.400910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.400877 5058 cpu_manager.go:414] "containerMap: removing container" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" Oct 30 05:34:45.401092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.401078 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" Oct 30 05:34:45.401450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.401432 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cec3e6f-9968-4ac4-94a2-0bc4b5b5d049" containerName="istio-proxy" Oct 30 05:34:45.445643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.445609 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:34:45.445959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.445943 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.469180 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.469148 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5b27e\"" Oct 30 05:34:45.488759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.488675 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5b27e-dockercfg-g2xz4\"" Oct 30 05:34:45.576339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.576295 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:34:45.593011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.592970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48ln\" (UniqueName: \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.593406 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.593363 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.627877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.627841 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.628106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.628079 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:34:45.695276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.695240 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48ln\" (UniqueName: \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.695276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.695284 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfh5z\" (UniqueName: \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.695592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.695315 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.695592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.695389 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.696049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.696022 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.718527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.718493 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48ln\" (UniqueName: \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.796878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.796790 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfh5z\" (UniqueName: \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.796878 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.796840 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.797647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.797625 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.818859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.818821 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfh5z\" (UniqueName: \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\") pod \"payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:45.899463 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.899432 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:45.950679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:45.950642 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:46.259377 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:34:46.259343 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eca7964_ae42_4d06_8188_86688f231077.slice/crio-8f76b35adb48c65522dfb1dbf47f54f49b06e192a63f2191af2b76c6d425d24f WatchSource:0}: Error finding container 8f76b35adb48c65522dfb1dbf47f54f49b06e192a63f2191af2b76c6d425d24f: Status 404 returned error can't find the container with id 8f76b35adb48c65522dfb1dbf47f54f49b06e192a63f2191af2b76c6d425d24f Oct 30 05:34:46.261348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:46.261324 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:34:46.278057 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:34:46.278018 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736c65c1_cc6f_4aed_92d4_0a17c53bf0cd.slice/crio-229cc76dc45aaa7e5a8034aa9de0a9ea0092b1589b60ce365bf80099963cc323 WatchSource:0}: Error finding container 229cc76dc45aaa7e5a8034aa9de0a9ea0092b1589b60ce365bf80099963cc323: Status 404 returned error can't find the container with id 229cc76dc45aaa7e5a8034aa9de0a9ea0092b1589b60ce365bf80099963cc323 Oct 30 05:34:46.281822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:46.281793 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:34:47.261255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.261186 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" event={"ID":"9eca7964-ae42-4d06-8188-86688f231077","Type":"ContainerStarted","Data":"b53a7e684a549c21007cae4b03508571674e2716a1bfdbbfd353938d6dd1d9e9"} Oct 30 05:34:47.261767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.261282 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" event={"ID":"9eca7964-ae42-4d06-8188-86688f231077","Type":"ContainerStarted","Data":"8f76b35adb48c65522dfb1dbf47f54f49b06e192a63f2191af2b76c6d425d24f"} Oct 30 05:34:47.261767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.261320 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:47.263271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.263206 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" event={"ID":"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd","Type":"ContainerStarted","Data":"6845ecf491e07dabf3912fbb75b10975b6b3ca9803434e6736c63828cce0026a"} Oct 30 05:34:47.263420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.263298 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" event={"ID":"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd","Type":"ContainerStarted","Data":"229cc76dc45aaa7e5a8034aa9de0a9ea0092b1589b60ce365bf80099963cc323"} Oct 30 05:34:47.263420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.263359 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:47.314708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.314657 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" podStartSLOduration=2.314642188 podStartE2EDuration="2.314642188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:47.310238223 +0000 UTC m=+3060.279733215" watchObservedRunningTime="2026-10-30 05:34:47.314642188 +0000 UTC m=+3060.284137176" Oct 30 05:34:47.366394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:47.366341 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" podStartSLOduration=2.366327 podStartE2EDuration="2.366327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:47.360796772 +0000 UTC m=+3060.330291760" watchObservedRunningTime="2026-10-30 05:34:47.366327 +0000 UTC m=+3060.335822003" Oct 30 05:34:49.194150 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:49.193304 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:34:53.983484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:53.983432 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerName="main" containerID="cri-o://71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7" gracePeriod=45 Oct 30 05:34:54.336353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.336324 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:34:54.447444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447406 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447555 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447602 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\" (UniqueName: \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447706 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447816 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:54.447860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447840 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447900 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\") pod \"b1314cab-6d61-40cf-98f7-be85d74b1f39\" (UID: \"b1314cab-6d61-40cf-98f7-be85d74b1f39\") " Oct 30 05:34:54.447985 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.447952 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:54.448121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.448089 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:54.448564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.448544 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.448666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.448566 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.448666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.448579 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.450039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.450015 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:54.450170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.450055 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:54.450170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.450133 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq" pod "b1314cab-6d61-40cf-98f7-be85d74b1f39" (UID: "b1314cab-6d61-40cf-98f7-be85d74b1f39"). InnerVolumeSpecName "kube-api-access-gqzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:54.558685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.558595 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqzbq\" (UniqueName: \"kubernetes.io/projected/b1314cab-6d61-40cf-98f7-be85d74b1f39-kube-api-access-gqzbq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.558685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.558633 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1314cab-6d61-40cf-98f7-be85d74b1f39-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.558685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.558648 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1314cab-6d61-40cf-98f7-be85d74b1f39-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:54.655353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.655315 5058 generic.go:356] "Generic (PLEG): container finished" podID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerID="71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7" exitCode=0 Oct 30 05:34:54.655546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.655383 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" Oct 30 05:34:54.655546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.655394 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" event={"ID":"b1314cab-6d61-40cf-98f7-be85d74b1f39","Type":"ContainerDied","Data":"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7"} Oct 30 05:34:54.655546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.655439 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk" event={"ID":"b1314cab-6d61-40cf-98f7-be85d74b1f39","Type":"ContainerDied","Data":"4dc80b57d3fb1afb0a1402d4c2a5136dbd2c7969826a8dbdd79685c77f71fadb"} Oct 30 05:34:54.655546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.655460 5058 scope.go:122] "RemoveContainer" containerID="71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7" Oct 30 05:34:54.669012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.668989 5058 scope.go:122] "RemoveContainer" containerID="71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7" Oct 30 05:34:54.669444 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:34:54.669412 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7\": container with ID starting with 71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7 not found: ID does not exist" containerID="71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7" Oct 30 05:34:54.669565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.669452 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7"} err="failed to get container status \"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7\": rpc error: code = NotFound desc = could not find container \"71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7\": container with ID starting with 71f6fe57e2566b1165fd925155c991ece158a1659d8098dd047e3346081f7bf7 not found: ID does not exist" Oct 30 05:34:54.716355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.716315 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:34:54.734097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:54.734053 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/rate-test-model-674082-kserve-fb558df85-gc4pk"] Oct 30 05:34:55.166311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:55.166260 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" path="/var/lib/kubelet/pods/b1314cab-6d61-40cf-98f7-be85d74b1f39/volumes" Oct 30 05:34:56.162149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.162117 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:34:56.163115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.163096 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerName="main" Oct 30 05:34:56.163185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.163116 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerName="main" Oct 30 05:34:56.163257 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.163246 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1314cab-6d61-40cf-98f7-be85d74b1f39" containerName="main" Oct 30 05:34:56.169272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.169202 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.186274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.186241 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:34:56.216574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.216541 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9bg\" (UniqueName: \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.216574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.216579 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.218212 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.218193 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec1091\"" Oct 30 05:34:56.218949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.218933 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec1091-dockercfg-rrzfr\"" Oct 30 05:34:56.318875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.318838 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9bg\" (UniqueName: \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.319151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.318881 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.319852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.319823 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.340859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.340832 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9bg\" (UniqueName: \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\") pod \"payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.367435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.367400 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:34:56.367638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.367598 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" podUID="28b0ee62-4ec2-47db-820e-531cc6eee35b" containerName="maas-api" containerID="cri-o://0ef8a6e8c6a8958d266b474ffa5336d14d34b428c558897338ec37a29d1953b3" gracePeriod=30 Oct 30 05:34:56.486019 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.485995 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:56.830384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.830008 5058 generic.go:356] "Generic (PLEG): container finished" podID="28b0ee62-4ec2-47db-820e-531cc6eee35b" containerID="0ef8a6e8c6a8958d266b474ffa5336d14d34b428c558897338ec37a29d1953b3" exitCode=0 Oct 30 05:34:56.830384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.830286 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" event={"ID":"28b0ee62-4ec2-47db-820e-531cc6eee35b","Type":"ContainerDied","Data":"0ef8a6e8c6a8958d266b474ffa5336d14d34b428c558897338ec37a29d1953b3"} Oct 30 05:34:56.983176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:56.983138 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:34:57.014090 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:34:57.014054 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc4ae68_d95b_4d50_9e6b_979b3bc7b04a.slice/crio-3e80451d92a220f8184c1e9685b646390d174025d20cbcdca8889311a68d32ed WatchSource:0}: Error finding container 3e80451d92a220f8184c1e9685b646390d174025d20cbcdca8889311a68d32ed: Status 404 returned error can't find the container with id 3e80451d92a220f8184c1e9685b646390d174025d20cbcdca8889311a68d32ed Oct 30 05:34:57.015685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.015661 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:34:57.164567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.164237 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\") pod \"28b0ee62-4ec2-47db-820e-531cc6eee35b\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " Oct 30 05:34:57.165307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.164505 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\") pod \"28b0ee62-4ec2-47db-820e-531cc6eee35b\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " Oct 30 05:34:57.165685 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.165659 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\" (UniqueName: \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\") pod \"28b0ee62-4ec2-47db-820e-531cc6eee35b\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " Oct 30 05:34:57.165829 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.165813 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\") pod \"28b0ee62-4ec2-47db-820e-531cc6eee35b\" (UID: \"28b0ee62-4ec2-47db-820e-531cc6eee35b\") " Oct 30 05:34:57.167693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.165289 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca" pod "28b0ee62-4ec2-47db-820e-531cc6eee35b" (UID: "28b0ee62-4ec2-47db-820e-531cc6eee35b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:57.170171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.170147 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9" pod "28b0ee62-4ec2-47db-820e-531cc6eee35b" (UID: "28b0ee62-4ec2-47db-820e-531cc6eee35b"). InnerVolumeSpecName "kube-api-access-lk5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:57.170608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.170265 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs" pod "28b0ee62-4ec2-47db-820e-531cc6eee35b" (UID: "28b0ee62-4ec2-47db-820e-531cc6eee35b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:57.174292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.173208 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls" pod "28b0ee62-4ec2-47db-820e-531cc6eee35b" (UID: "28b0ee62-4ec2-47db-820e-531cc6eee35b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:57.268322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.268117 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.268322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.268157 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0ee62-4ec2-47db-820e-531cc6eee35b-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.268322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.268172 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0ee62-4ec2-47db-820e-531cc6eee35b-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.268322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.268187 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk5r9\" (UniqueName: \"kubernetes.io/projected/28b0ee62-4ec2-47db-820e-531cc6eee35b-kube-api-access-lk5r9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.841359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.841317 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" event={"ID":"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a","Type":"ContainerStarted","Data":"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4"} Oct 30 05:34:57.841617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.841385 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" event={"ID":"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a","Type":"ContainerStarted","Data":"3e80451d92a220f8184c1e9685b646390d174025d20cbcdca8889311a68d32ed"} Oct 30 05:34:57.841617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.841432 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:34:57.843555 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.843497 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" Oct 30 05:34:57.843555 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.843536 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg" event={"ID":"28b0ee62-4ec2-47db-820e-531cc6eee35b","Type":"ContainerDied","Data":"9479825776f755eca02e2f99328d5de8f5eee41a45f9455e99ed2b87d606e125"} Oct 30 05:34:57.843775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.843606 5058 scope.go:122] "RemoveContainer" containerID="0ef8a6e8c6a8958d266b474ffa5336d14d34b428c558897338ec37a29d1953b3" Oct 30 05:34:57.887348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.887294 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" podStartSLOduration=1.8872783869999998 podStartE2EDuration="1.887278387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:57.884787665 +0000 UTC m=+3070.854282653" watchObservedRunningTime="2026-10-30 05:34:57.887278387 +0000 UTC m=+3070.856773375" Oct 30 05:34:57.931547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.931512 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:34:57.939901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:57.939870 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-262c10-7d885889f-wz2tg"] Oct 30 05:34:58.274104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:58.274069 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:34:58.274657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:58.274151 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:34:59.166812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:34:59.166775 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b0ee62-4ec2-47db-820e-531cc6eee35b" path="/var/lib/kubelet/pods/28b0ee62-4ec2-47db-820e-531cc6eee35b/volumes" Oct 30 05:35:05.855060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:05.855026 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:35:05.855636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:05.855341 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" containerID="cri-o://e45dee61b8405f0dc5bbd557468a4ef55925e7caa877e6d92068fef69bc93230" gracePeriod=30 Oct 30 05:35:08.856346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:08.856314 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:35:09.352258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:09.352183 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:35:09.352435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:09.352264 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:35:11.136814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.136780 5058 generic.go:356] "Generic (PLEG): container finished" podID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerID="e45dee61b8405f0dc5bbd557468a4ef55925e7caa877e6d92068fef69bc93230" exitCode=0 Oct 30 05:35:11.137186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.136833 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" event={"ID":"0b6f7431-234b-4cda-85cb-6b2de9439561","Type":"ContainerDied","Data":"e45dee61b8405f0dc5bbd557468a4ef55925e7caa877e6d92068fef69bc93230"} Oct 30 05:35:11.173132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.173097 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:35:11.204647 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204608 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.204828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204678 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.204828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204719 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.204828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204770 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204918 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.204959 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205079 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205120 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205178 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\") pod \"0b6f7431-234b-4cda-85cb-6b2de9439561\" (UID: \"0b6f7431-234b-4cda-85cb-6b2de9439561\") " Oct 30 05:35:11.205359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205319 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.205787 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205712 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.205919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205788 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:11.205919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.205855 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.206703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206170 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.206933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206905 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f7431-234b-4cda-85cb-6b2de9439561-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.206933 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206935 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.207155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206948 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.207155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206962 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.207155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.206979 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.207501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.207475 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.207583 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.207564 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.207678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.207664 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:11.209009 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.208988 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r" pod "0b6f7431-234b-4cda-85cb-6b2de9439561" (UID: "0b6f7431-234b-4cda-85cb-6b2de9439561"). InnerVolumeSpecName "kube-api-access-w4n2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.309412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.309372 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.309412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.309412 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.309605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.309428 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f7431-234b-4cda-85cb-6b2de9439561-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.309605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:11.309442 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4n2r\" (UniqueName: \"kubernetes.io/projected/0b6f7431-234b-4cda-85cb-6b2de9439561-kube-api-access-w4n2r\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:12.148510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:12.148476 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" Oct 30 05:35:12.149020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:12.148501 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l" event={"ID":"0b6f7431-234b-4cda-85cb-6b2de9439561","Type":"ContainerDied","Data":"c5f0c4890e61baa54d66d28b2a696612f46a1f368e478f439a418fb17430bf45"} Oct 30 05:35:12.149020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:12.148581 5058 scope.go:122] "RemoveContainer" containerID="e45dee61b8405f0dc5bbd557468a4ef55925e7caa877e6d92068fef69bc93230" Oct 30 05:35:12.260240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:12.259299 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:35:12.271614 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:12.270333 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l"] Oct 30 05:35:13.172850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:13.172806 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" path="/var/lib/kubelet/pods/0b6f7431-234b-4cda-85cb-6b2de9439561/volumes" Oct 30 05:35:14.223290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.222161 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:35:14.223290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223154 5058 cpu_manager.go:414] "containerMap: removing container" podUID="28b0ee62-4ec2-47db-820e-531cc6eee35b" containerName="maas-api" Oct 30 05:35:14.223290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223171 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b0ee62-4ec2-47db-820e-531cc6eee35b" containerName="maas-api" Oct 30 05:35:14.223290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223203 5058 cpu_manager.go:414] "containerMap: removing container" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" Oct 30 05:35:14.223290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223208 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" Oct 30 05:35:14.223850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223392 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b0ee62-4ec2-47db-820e-531cc6eee35b" containerName="maas-api" Oct 30 05:35:14.223850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.223410 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6f7431-234b-4cda-85cb-6b2de9439561" containerName="istio-proxy" Oct 30 05:35:14.281571 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.281526 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:35:14.281815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.281686 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.304301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.304272 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7a4600-openshift-default-dockercfg-nfmzs\"" Oct 30 05:35:14.380573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.380539 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.380806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.380791 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.380923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.380909 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381061 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381187 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381337 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381464 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381636 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.381824 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.381812 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrdx\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483205 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483387 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrdx\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483492 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483523 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483547 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483596 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483623 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483679 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483759 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483771 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.483936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.483852 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.484154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.484120 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.484280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.484175 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.484280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.484270 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.485906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.485885 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.486392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.486368 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.510049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.509875 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrdx\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.510284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.510196 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\") pod \"e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:14.609858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:14.608533 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:15.362101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:15.362067 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:35:15.365871 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:35:15.365833 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61810822_ab44_48db_8060_240e7c848ca0.slice/crio-4ec2a6d6807c03e2c6588cb62512d90ddf969a6641f32e7a334d156059ded0dc WatchSource:0}: Error finding container 4ec2a6d6807c03e2c6588cb62512d90ddf969a6641f32e7a334d156059ded0dc: Status 404 returned error can't find the container with id 4ec2a6d6807c03e2c6588cb62512d90ddf969a6641f32e7a334d156059ded0dc Oct 30 05:35:15.368777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:15.368746 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:15.368860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:15.368813 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:15.368860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:15.368843 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:16.196209 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.196173 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerStarted","Data":"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200"} Oct 30 05:35:16.196407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.196257 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerStarted","Data":"4ec2a6d6807c03e2c6588cb62512d90ddf969a6641f32e7a334d156059ded0dc"} Oct 30 05:35:16.286043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.285987 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podStartSLOduration=2.28597399 podStartE2EDuration="2.28597399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:16.281988377 +0000 UTC m=+3089.251483367" watchObservedRunningTime="2026-10-30 05:35:16.28597399 +0000 UTC m=+3089.255469052" Oct 30 05:35:16.610141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.610049 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:16.616407 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.616373 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:16.955195 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.955158 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:35:16.973962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:16.973928 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:35:17.294832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:17.293953 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:17.295007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:17.294982 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:35:19.196805 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:35:19.196760 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:35:21.970751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:21.967008 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" containerID="cri-o://9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c" gracePeriod=25 Oct 30 05:35:21.975122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:21.974876 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" containerID="cri-o://96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef" gracePeriod=25 Oct 30 05:35:23.453295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.453262 5058 generic.go:356] "Generic (PLEG): container finished" podID="c43bbdb4-3917-460e-8c18-d81926232c43" containerID="d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f" exitCode=0 Oct 30 05:35:23.453800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.453336 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerDied","Data":"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f"} Oct 30 05:35:23.453898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.453869 5058 scope.go:122] "RemoveContainer" containerID="d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f" Oct 30 05:35:23.455888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.455859 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:23.455962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.455932 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:23.456004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:23.455969 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:24.465123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:24.465078 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerStarted","Data":"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001"} Oct 30 05:35:25.146242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:25.142726 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:35:25.146242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:25.142888 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:35:25.146242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:25.142947 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:35:26.146243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:26.144693 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:35:26.146243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:26.144763 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:35:26.676914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:26.676870 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:26.677104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:26.676870 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:27.146243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:27.142686 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:35:27.146243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:27.142754 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:35:28.151295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:28.151045 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:35:28.151295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:28.151107 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:35:29.151897 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:29.151855 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 05:35:29.152457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:29.152323 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 05:35:30.147507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:30.147476 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:35:30.148029 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:30.147994 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:35:30.149165 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:30.149140 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:35:36.676816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:36.676767 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:36.677301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:36.676815 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:42.304834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.304799 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:35:42.340465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.340435 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:35:42.452292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.452258 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\" (UniqueName: \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\") pod \"5c7e3e12-78ac-427c-9790-004b30634d4d\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " Oct 30 05:35:42.452556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.452332 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\" (UniqueName: \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\") pod \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " Oct 30 05:35:42.452556 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.452441 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\") pod \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\" (UID: \"5ef450b5-d521-44f0-b2b0-4808c3f503d8\") " Oct 30 05:35:42.452698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.452591 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\") pod \"5c7e3e12-78ac-427c-9790-004b30634d4d\" (UID: \"5c7e3e12-78ac-427c-9790-004b30634d4d\") " Oct 30 05:35:42.453044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.452999 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume" pod "5ef450b5-d521-44f0-b2b0-4808c3f503d8" (UID: "5ef450b5-d521-44f0-b2b0-4808c3f503d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:42.453151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.453019 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume" pod "5c7e3e12-78ac-427c-9790-004b30634d4d" (UID: "5c7e3e12-78ac-427c-9790-004b30634d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:42.453536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.453509 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef450b5-d521-44f0-b2b0-4808c3f503d8-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:42.453643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.453541 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7e3e12-78ac-427c-9790-004b30634d4d-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:42.454733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.454709 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp" pod "5ef450b5-d521-44f0-b2b0-4808c3f503d8" (UID: "5ef450b5-d521-44f0-b2b0-4808c3f503d8"). InnerVolumeSpecName "kube-api-access-tjxsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:42.454877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.454855 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv" pod "5c7e3e12-78ac-427c-9790-004b30634d4d" (UID: "5c7e3e12-78ac-427c-9790-004b30634d4d"). InnerVolumeSpecName "kube-api-access-l7gtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:42.555601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.555508 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7gtv\" (UniqueName: \"kubernetes.io/projected/5c7e3e12-78ac-427c-9790-004b30634d4d-kube-api-access-l7gtv\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:42.555601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.555543 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxsp\" (UniqueName: \"kubernetes.io/projected/5ef450b5-d521-44f0-b2b0-4808c3f503d8-kube-api-access-tjxsp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:42.682676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.682640 5058 generic.go:356] "Generic (PLEG): container finished" podID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerID="9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c" exitCode=0 Oct 30 05:35:42.682865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.682693 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" Oct 30 05:35:42.682865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.682724 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" event={"ID":"5c7e3e12-78ac-427c-9790-004b30634d4d","Type":"ContainerDied","Data":"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c"} Oct 30 05:35:42.682865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.682826 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt" event={"ID":"5c7e3e12-78ac-427c-9790-004b30634d4d","Type":"ContainerDied","Data":"2b8592ee55835e7662f0d6a42989468db58f3351f1dedb693b69461d1a1fc606"} Oct 30 05:35:42.682865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.682855 5058 scope.go:122] "RemoveContainer" containerID="9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c" Oct 30 05:35:42.684301 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.684277 5058 generic.go:356] "Generic (PLEG): container finished" podID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerID="96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef" exitCode=0 Oct 30 05:35:42.684423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.684326 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" Oct 30 05:35:42.684423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.684373 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" event={"ID":"5ef450b5-d521-44f0-b2b0-4808c3f503d8","Type":"ContainerDied","Data":"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef"} Oct 30 05:35:42.684423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.684419 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8" event={"ID":"5ef450b5-d521-44f0-b2b0-4808c3f503d8","Type":"ContainerDied","Data":"41583137ff38fc3cd68020a934205694ef876ec915c3a0515a42ca77a3f3c0e6"} Oct 30 05:35:42.694700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.694680 5058 scope.go:122] "RemoveContainer" containerID="9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c" Oct 30 05:35:42.695037 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:35:42.695006 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c\": container with ID starting with 9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c not found: ID does not exist" containerID="9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c" Oct 30 05:35:42.695159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.695050 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c"} err="failed to get container status \"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c\": rpc error: code = NotFound desc = could not find container \"9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c\": container with ID starting with 9cc8bceb808b25d67db2a415ad8cf87e188d32e8274fd3cbfeddcea4d20ebb9c not found: ID does not exist" Oct 30 05:35:42.695159 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.695076 5058 scope.go:122] "RemoveContainer" containerID="96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef" Oct 30 05:35:42.706111 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.706088 5058 scope.go:122] "RemoveContainer" containerID="96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef" Oct 30 05:35:42.706533 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:35:42.706512 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef\": container with ID starting with 96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef not found: ID does not exist" containerID="96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef" Oct 30 05:35:42.706617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.706546 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef"} err="failed to get container status \"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef\": rpc error: code = NotFound desc = could not find container \"96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef\": container with ID starting with 96cf59e727476f136485ac869734004bad74cf9096caa8d8244ccff9e07ca6ef not found: ID does not exist" Oct 30 05:35:42.743570 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.743534 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:35:42.747709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.747680 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8"] Oct 30 05:35:42.788544 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.788507 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:35:42.796964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:42.796926 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt"] Oct 30 05:35:43.164046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:43.164008 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" path="/var/lib/kubelet/pods/5c7e3e12-78ac-427c-9790-004b30634d4d/volumes" Oct 30 05:35:43.164733 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:43.164698 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" path="/var/lib/kubelet/pods/5ef450b5-d521-44f0-b2b0-4808c3f503d8/volumes" Oct 30 05:35:46.035667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.035625 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 05:35:46.036338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.035905 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" podUID="f6e2856c-2432-472d-9930-2a7ef63e75ab" containerName="maas-api" containerID="cri-o://e2c3aa3b48bc3d0527754c8d1fec84f25a475355267ff6f07ce4848aef28ffe6" gracePeriod=30 Oct 30 05:35:46.742760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.742715 5058 generic.go:356] "Generic (PLEG): container finished" podID="f6e2856c-2432-472d-9930-2a7ef63e75ab" containerID="e2c3aa3b48bc3d0527754c8d1fec84f25a475355267ff6f07ce4848aef28ffe6" exitCode=0 Oct 30 05:35:46.743059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.743023 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" event={"ID":"f6e2856c-2432-472d-9930-2a7ef63e75ab","Type":"ContainerDied","Data":"e2c3aa3b48bc3d0527754c8d1fec84f25a475355267ff6f07ce4848aef28ffe6"} Oct 30 05:35:46.814837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.814807 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 05:35:46.964269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.964210 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\") pod \"f6e2856c-2432-472d-9930-2a7ef63e75ab\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " Oct 30 05:35:46.964470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.964366 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\") pod \"f6e2856c-2432-472d-9930-2a7ef63e75ab\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " Oct 30 05:35:46.964470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.964445 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\") pod \"f6e2856c-2432-472d-9930-2a7ef63e75ab\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " Oct 30 05:35:46.964611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.964470 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") pod \"f6e2856c-2432-472d-9930-2a7ef63e75ab\" (UID: \"f6e2856c-2432-472d-9930-2a7ef63e75ab\") " Oct 30 05:35:46.964813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.964773 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca" pod "f6e2856c-2432-472d-9930-2a7ef63e75ab" (UID: "f6e2856c-2432-472d-9930-2a7ef63e75ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:46.965175 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.965147 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e2856c-2432-472d-9930-2a7ef63e75ab-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:46.966797 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.966768 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls" pod "f6e2856c-2432-472d-9930-2a7ef63e75ab" (UID: "f6e2856c-2432-472d-9930-2a7ef63e75ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:46.966961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.966832 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs" pod "f6e2856c-2432-472d-9930-2a7ef63e75ab" (UID: "f6e2856c-2432-472d-9930-2a7ef63e75ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:46.966961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:46.966874 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw" pod "f6e2856c-2432-472d-9930-2a7ef63e75ab" (UID: "f6e2856c-2432-472d-9930-2a7ef63e75ab"). InnerVolumeSpecName "kube-api-access-59rfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:47.066761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.066660 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:47.066761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.066700 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e2856c-2432-472d-9930-2a7ef63e75ab-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:47.066761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.066712 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/f6e2856c-2432-472d-9930-2a7ef63e75ab-kube-api-access-59rfw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:47.752037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.752002 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" Oct 30 05:35:47.752235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.752038 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt" event={"ID":"f6e2856c-2432-472d-9930-2a7ef63e75ab","Type":"ContainerDied","Data":"79c80fd2f590c993266f80ef36a65b79de020395057d040a7dd032b648a922ca"} Oct 30 05:35:47.752235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.752102 5058 scope.go:122] "RemoveContainer" containerID="e2c3aa3b48bc3d0527754c8d1fec84f25a475355267ff6f07ce4848aef28ffe6" Oct 30 05:35:47.810364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.810323 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 05:35:47.823858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:47.823823 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-83e83e-776f9c58b6-2xmgt"] Oct 30 05:35:49.179196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:49.179160 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e2856c-2432-472d-9930-2a7ef63e75ab" path="/var/lib/kubelet/pods/f6e2856c-2432-472d-9930-2a7ef63e75ab/volumes" Oct 30 05:35:49.203463 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:35:49.203422 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:35:50.566286 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:50.566163 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:35:50.566743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:50.566409 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" podUID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" containerName="maas-api" containerID="cri-o://8ca3c0333908de3dfc88ebd4087588122aa46c3d48bd45740ef314acc80e0dc8" gracePeriod=30 Oct 30 05:35:50.784775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:50.784697 5058 generic.go:356] "Generic (PLEG): container finished" podID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" containerID="8ca3c0333908de3dfc88ebd4087588122aa46c3d48bd45740ef314acc80e0dc8" exitCode=0 Oct 30 05:35:50.784775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:50.784743 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" event={"ID":"dabfd824-40fe-45f2-86cf-acb23b2f9d07","Type":"ContainerDied","Data":"8ca3c0333908de3dfc88ebd4087588122aa46c3d48bd45740ef314acc80e0dc8"} Oct 30 05:35:51.147242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.145607 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:35:51.299063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.298943 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:35:51.299268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.299237 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerName="maas-api" containerID="cri-o://7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4" gracePeriod=30 Oct 30 05:35:51.306077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.306043 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\") pod \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " Oct 30 05:35:51.306592 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.306567 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\") pod \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " Oct 30 05:35:51.306825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.306807 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\" (UniqueName: \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\") pod \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " Oct 30 05:35:51.306902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.306842 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\") pod \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\" (UID: \"dabfd824-40fe-45f2-86cf-acb23b2f9d07\") " Oct 30 05:35:51.308187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.308162 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca" pod "dabfd824-40fe-45f2-86cf-acb23b2f9d07" (UID: "dabfd824-40fe-45f2-86cf-acb23b2f9d07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:51.315822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.315430 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls" pod "dabfd824-40fe-45f2-86cf-acb23b2f9d07" (UID: "dabfd824-40fe-45f2-86cf-acb23b2f9d07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:51.335044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.335011 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n" pod "dabfd824-40fe-45f2-86cf-acb23b2f9d07" (UID: "dabfd824-40fe-45f2-86cf-acb23b2f9d07"). InnerVolumeSpecName "kube-api-access-7xc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:51.335194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.335150 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs" pod "dabfd824-40fe-45f2-86cf-acb23b2f9d07" (UID: "dabfd824-40fe-45f2-86cf-acb23b2f9d07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:51.341455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.341420 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.219:8443/health\": dial tcp 10.132.0.219:8443: connect: connection refused" Oct 30 05:35:51.409638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.409599 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xc7n\" (UniqueName: \"kubernetes.io/projected/dabfd824-40fe-45f2-86cf-acb23b2f9d07-kube-api-access-7xc7n\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:51.409638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.409639 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:51.409848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.409654 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabfd824-40fe-45f2-86cf-acb23b2f9d07-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:51.409848 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.409670 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabfd824-40fe-45f2-86cf-acb23b2f9d07-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:51.651723 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.651697 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:35:51.793790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.793754 5058 generic.go:356] "Generic (PLEG): container finished" podID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerID="7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4" exitCode=0 Oct 30 05:35:51.793991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.793804 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" Oct 30 05:35:51.793991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.793838 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" event={"ID":"f5a6c749-7d46-413a-a440-6d6f579f0a7e","Type":"ContainerDied","Data":"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4"} Oct 30 05:35:51.793991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.793895 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799" event={"ID":"f5a6c749-7d46-413a-a440-6d6f579f0a7e","Type":"ContainerDied","Data":"83b7415f61117a69ac2be88d7fea7f69c8dc7b54e4f563cfe9e44bc923931fb2"} Oct 30 05:35:51.793991 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.793924 5058 scope.go:122] "RemoveContainer" containerID="7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4" Oct 30 05:35:51.795503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.795481 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" event={"ID":"dabfd824-40fe-45f2-86cf-acb23b2f9d07","Type":"ContainerDied","Data":"f95b8fc77170f9b6dacffba4f6ce632772146cb2aaa6683bf9425e143c8aebc7"} Oct 30 05:35:51.795503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.795494 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf" Oct 30 05:35:51.806114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.806095 5058 scope.go:122] "RemoveContainer" containerID="7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4" Oct 30 05:35:51.806448 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:35:51.806426 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4\": container with ID starting with 7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4 not found: ID does not exist" containerID="7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4" Oct 30 05:35:51.806536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.806463 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4"} err="failed to get container status \"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4\": rpc error: code = NotFound desc = could not find container \"7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4\": container with ID starting with 7a61274c47312b18037e69d38fbd01edd24aa86f5a331a4e3ba17319eb85d8c4 not found: ID does not exist" Oct 30 05:35:51.806536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.806489 5058 scope.go:122] "RemoveContainer" containerID="8ca3c0333908de3dfc88ebd4087588122aa46c3d48bd45740ef314acc80e0dc8" Oct 30 05:35:51.821384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.821097 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\") pod \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " Oct 30 05:35:51.821384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.821166 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\") pod \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " Oct 30 05:35:51.821384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.821197 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\" (UniqueName: \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\") pod \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " Oct 30 05:35:51.821384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.821349 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\") pod \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\" (UID: \"f5a6c749-7d46-413a-a440-6d6f579f0a7e\") " Oct 30 05:35:51.822467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.822436 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca" pod "f5a6c749-7d46-413a-a440-6d6f579f0a7e" (UID: "f5a6c749-7d46-413a-a440-6d6f579f0a7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:51.943579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.925139 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a6c749-7d46-413a-a440-6d6f579f0a7e-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:51.999706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:51.999589 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:35:52.097607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.097540 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw" pod "f5a6c749-7d46-413a-a440-6d6f579f0a7e" (UID: "f5a6c749-7d46-413a-a440-6d6f579f0a7e"). InnerVolumeSpecName "kube-api-access-2b2xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:52.107559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.107474 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs" pod "f5a6c749-7d46-413a-a440-6d6f579f0a7e" (UID: "f5a6c749-7d46-413a-a440-6d6f579f0a7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:52.112461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.112362 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-674082-7d4d69549c-whfbf"] Oct 30 05:35:52.113384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.113343 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls" pod "f5a6c749-7d46-413a-a440-6d6f579f0a7e" (UID: "f5a6c749-7d46-413a-a440-6d6f579f0a7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:52.164266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.163912 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.164266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.163949 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a6c749-7d46-413a-a440-6d6f579f0a7e-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.164266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.163964 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2xw\" (UniqueName: \"kubernetes.io/projected/f5a6c749-7d46-413a-a440-6d6f579f0a7e-kube-api-access-2b2xw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.497449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.497413 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:35:52.506430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:52.505660 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d21ab-5b6fc56bcc-ld799"] Oct 30 05:35:53.163357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:53.163325 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" path="/var/lib/kubelet/pods/dabfd824-40fe-45f2-86cf-acb23b2f9d07/volumes" Oct 30 05:35:53.163864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:53.163849 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" path="/var/lib/kubelet/pods/f5a6c749-7d46-413a-a440-6d6f579f0a7e/volumes" Oct 30 05:35:59.697018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:59.696972 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:35:59.697573 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:35:59.697188 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" containerID="cri-o://fbd9a79e143e2b6571a2c92d48bb0cea614858a5cad563d5ddc33632c149b002" gracePeriod=30 Oct 30 05:36:00.937842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.937808 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:36:00.938348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.938024 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" containerID="cri-o://1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898" gracePeriod=30 Oct 30 05:36:00.952085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.952034 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:36:00.953341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.952450 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" containerID="cri-o://d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9" gracePeriod=30 Oct 30 05:36:00.959664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.959166 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:36:00.959664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.959466 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" containerID="cri-o://b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9" gracePeriod=30 Oct 30 05:36:00.967478 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.967435 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:36:00.967726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:00.967697 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" containerID="cri-o://36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc" gracePeriod=30 Oct 30 05:36:04.970261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:04.970172 5058 generic.go:356] "Generic (PLEG): container finished" podID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerID="fbd9a79e143e2b6571a2c92d48bb0cea614858a5cad563d5ddc33632c149b002" exitCode=0 Oct 30 05:36:04.970261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:04.970248 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" event={"ID":"cd1c7973-bf36-4bac-94de-1e67c84ad104","Type":"ContainerDied","Data":"fbd9a79e143e2b6571a2c92d48bb0cea614858a5cad563d5ddc33632c149b002"} Oct 30 05:36:05.021153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.021123 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:36:05.075199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075165 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075242 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075443 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075481 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075575 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075611 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075689 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:05.075792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075702 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075849 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075834 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.075975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075938 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.076042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075995 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\") pod \"cd1c7973-bf36-4bac-94de-1e67c84ad104\" (UID: \"cd1c7973-bf36-4bac-94de-1e67c84ad104\") " Oct 30 05:36:05.076289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.075993 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.076289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076203 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.076289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076256 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.076912 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076891 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c7973-bf36-4bac-94de-1e67c84ad104-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.076954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076920 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.076954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076934 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.076954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076948 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.077056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.076962 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.077716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.077684 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.077908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.077892 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "kube-api-access-kq2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.078032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.078018 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.078114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.078094 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo" pod "cd1c7973-bf36-4bac-94de-1e67c84ad104" (UID: "cd1c7973-bf36-4bac-94de-1e67c84ad104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:05.184147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.184113 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.184147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.184147 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.184445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.184161 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.184445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.184236 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq2x2\" (UniqueName: \"kubernetes.io/projected/cd1c7973-bf36-4bac-94de-1e67c84ad104-kube-api-access-kq2x2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.232759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.232152 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:36:05.237368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:05.237276 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 05:36:06.000504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.000460 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" event={"ID":"cd1c7973-bf36-4bac-94de-1e67c84ad104","Type":"ContainerDied","Data":"1b74859fb2221191c3a7d7d107eb49ee537e7b3f8f23216cb45f535472ca7338"} Oct 30 05:36:06.001070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.000553 5058 scope.go:122] "RemoveContainer" containerID="fbd9a79e143e2b6571a2c92d48bb0cea614858a5cad563d5ddc33632c149b002" Oct 30 05:36:06.001070 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.000692 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj" Oct 30 05:36:06.238013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.237933 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:36:06.250752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.250720 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj"] Oct 30 05:36:06.282715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.282685 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:36:06.324859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.324822 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.324921 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325022 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325133 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325165 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325351 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325397 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.325843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.325678 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.326288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.326162 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.326288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.326190 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\") pod \"44504719-7e7d-43af-961c-f5966c48c968\" (UID: \"44504719-7e7d-43af-961c-f5966c48c968\") " Oct 30 05:36:06.327338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.326981 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.327338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.327014 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44504719-7e7d-43af-961c-f5966c48c968-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.327338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.327052 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.327338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.327117 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.327338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.327317 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.333418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.333343 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "kube-api-access-r4lbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.333418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.333381 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.333569 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.333473 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.334607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.334572 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token" pod "44504719-7e7d-43af-961c-f5966c48c968" (UID: "44504719-7e7d-43af-961c-f5966c48c968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.358408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.358006 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:36:06.387776 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.387748 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:36:06.391894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.391867 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:36:06.428384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428349 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.428585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428418 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.428585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428450 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.428585 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428532 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428600 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428633 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428667 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428705 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428734 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.428783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428772 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428895 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428913 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.428957 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429010 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429054 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.429117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429116 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.429482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429126 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.429482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429172 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.429482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429208 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\") pod \"41d0cb58-f059-4b15-83bd-6e37a907234c\" (UID: \"41d0cb58-f059-4b15-83bd-6e37a907234c\") " Oct 30 05:36:06.429482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429292 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\") pod \"7b57f9b3-b940-46df-8e3d-10509d96fab0\" (UID: \"7b57f9b3-b940-46df-8e3d-10509d96fab0\") " Oct 30 05:36:06.429482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429371 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.429788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429768 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.429971 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.430151 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.430166 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.430376 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.430541 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.430936 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.430554 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.431464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.431387 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "kube-api-access-rfqfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.431717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.431691 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.432510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432485 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfqfp\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-kube-api-access-rfqfp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432517 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432534 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4lbl\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-kube-api-access-r4lbl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432548 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432561 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432574 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432588 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432603 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432617 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432630 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44504719-7e7d-43af-961c-f5966c48c968-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432643 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432657 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432670 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d0cb58-f059-4b15-83bd-6e37a907234c-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432683 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432695 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57f9b3-b940-46df-8e3d-10509d96fab0-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432708 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432719 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432731 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432744 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44504719-7e7d-43af-961c-f5966c48c968-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.432882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432756 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44504719-7e7d-43af-961c-f5966c48c968-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.433445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.432880 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.433445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.433125 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "kube-api-access-rpsfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.433445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.433158 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.434901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.434874 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.445279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.436400 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token" pod "7b57f9b3-b940-46df-8e3d-10509d96fab0" (UID: "7b57f9b3-b940-46df-8e3d-10509d96fab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.445279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.439977 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy" pod "41d0cb58-f059-4b15-83bd-6e37a907234c" (UID: "41d0cb58-f059-4b15-83bd-6e37a907234c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.533990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.533900 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.533990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.533962 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534020 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534104 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534171 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.534269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534186 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534324 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534366 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534436 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534468 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.534523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534486 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.534742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534558 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\") pod \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\" (UID: \"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77\") " Oct 30 05:36:06.534799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534778 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.534861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.534788 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535310 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535337 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535354 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535365 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535379 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535392 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535405 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535419 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsfw\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-kube-api-access-rpsfw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535432 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57f9b3-b940-46df-8e3d-10509d96fab0-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535445 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d0cb58-f059-4b15-83bd-6e37a907234c-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.535482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.535457 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.536686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.536657 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "kube-api-access-47mfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.536873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.536853 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.536873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.536864 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.537082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.537060 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token" pod "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" (UID: "b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.638504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.637274 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.638504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.637313 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.638504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.637466 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.638504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:06.637480 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mfc\" (UniqueName: \"kubernetes.io/projected/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77-kube-api-access-47mfc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:07.010375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.010338 5058 generic.go:356] "Generic (PLEG): container finished" podID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerID="d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9" exitCode=0 Oct 30 05:36:07.010809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.010392 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" Oct 30 05:36:07.010809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.010418 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" event={"ID":"7b57f9b3-b940-46df-8e3d-10509d96fab0","Type":"ContainerDied","Data":"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9"} Oct 30 05:36:07.010809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.010477 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt" event={"ID":"7b57f9b3-b940-46df-8e3d-10509d96fab0","Type":"ContainerDied","Data":"74b1feb4d55ed625f46a0f83aca11ae1bd2cae274c6fe10bda3337201a22e9a7"} Oct 30 05:36:07.010809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.010497 5058 scope.go:122] "RemoveContainer" containerID="d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9" Oct 30 05:36:07.013200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.013177 5058 generic.go:356] "Generic (PLEG): container finished" podID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerID="1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898" exitCode=0 Oct 30 05:36:07.013329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.013245 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" Oct 30 05:36:07.013329 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.013278 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" event={"ID":"41d0cb58-f059-4b15-83bd-6e37a907234c","Type":"ContainerDied","Data":"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898"} Oct 30 05:36:07.013451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.013330 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5" event={"ID":"41d0cb58-f059-4b15-83bd-6e37a907234c","Type":"ContainerDied","Data":"7e55bc81502dd170409ef323ccb65d1d41c3796c8c8218ff039fe86b52216ec0"} Oct 30 05:36:07.014814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.014792 5058 generic.go:356] "Generic (PLEG): container finished" podID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerID="b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9" exitCode=0 Oct 30 05:36:07.014907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.014831 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" Oct 30 05:36:07.014907 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.014871 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" event={"ID":"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77","Type":"ContainerDied","Data":"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9"} Oct 30 05:36:07.014984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.014914 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr" event={"ID":"b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77","Type":"ContainerDied","Data":"c669fb96229868160f7d00934c47c1323f2f9406a842cff1f6d27f02932d94a9"} Oct 30 05:36:07.021816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.016471 5058 generic.go:356] "Generic (PLEG): container finished" podID="44504719-7e7d-43af-961c-f5966c48c968" containerID="36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc" exitCode=0 Oct 30 05:36:07.021816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.016531 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" event={"ID":"44504719-7e7d-43af-961c-f5966c48c968","Type":"ContainerDied","Data":"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc"} Oct 30 05:36:07.021816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.016552 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" Oct 30 05:36:07.021816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.016569 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz" event={"ID":"44504719-7e7d-43af-961c-f5966c48c968","Type":"ContainerDied","Data":"37c4a32a8ad3bd15e7683171135680fc978d5b52ecc67b11cee1a0cfdaae4f0d"} Oct 30 05:36:07.024497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.024478 5058 scope.go:122] "RemoveContainer" containerID="d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9" Oct 30 05:36:07.024819 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:07.024800 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9\": container with ID starting with d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9 not found: ID does not exist" containerID="d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9" Oct 30 05:36:07.024902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.024831 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9"} err="failed to get container status \"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9\": rpc error: code = NotFound desc = could not find container \"d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9\": container with ID starting with d06e168fdf3d26b889ae1a2a5dd0c851e84202ac259cfdb37fce9ce9206da3e9 not found: ID does not exist" Oct 30 05:36:07.024902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.024853 5058 scope.go:122] "RemoveContainer" containerID="1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898" Oct 30 05:36:07.038109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.038088 5058 scope.go:122] "RemoveContainer" containerID="1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898" Oct 30 05:36:07.038500 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:07.038475 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898\": container with ID starting with 1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898 not found: ID does not exist" containerID="1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898" Oct 30 05:36:07.038610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.038509 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898"} err="failed to get container status \"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898\": rpc error: code = NotFound desc = could not find container \"1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898\": container with ID starting with 1e897442d5435e49b67fa2136b0493e9f5507ac907d43a4a992a5eab4002d898 not found: ID does not exist" Oct 30 05:36:07.038610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.038534 5058 scope.go:122] "RemoveContainer" containerID="b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9" Oct 30 05:36:07.048887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.048867 5058 scope.go:122] "RemoveContainer" containerID="b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9" Oct 30 05:36:07.049245 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:07.049194 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9\": container with ID starting with b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9 not found: ID does not exist" containerID="b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9" Oct 30 05:36:07.049326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.049234 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9"} err="failed to get container status \"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9\": rpc error: code = NotFound desc = could not find container \"b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9\": container with ID starting with b2948496478d2b2703505e2e1d273a88863c2a2358487ef776065efcda6f02f9 not found: ID does not exist" Oct 30 05:36:07.049326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.049256 5058 scope.go:122] "RemoveContainer" containerID="36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc" Oct 30 05:36:07.059326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.059309 5058 scope.go:122] "RemoveContainer" containerID="36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc" Oct 30 05:36:07.059619 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:07.059594 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc\": container with ID starting with 36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc not found: ID does not exist" containerID="36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc" Oct 30 05:36:07.059697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.059624 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc"} err="failed to get container status \"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc\": rpc error: code = NotFound desc = could not find container \"36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc\": container with ID starting with 36da7ced27bbe166b8147038103df7458d5b4819fdd46a79d9a563d0cc9329cc not found: ID does not exist" Oct 30 05:36:07.068823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.068795 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:36:07.082507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.082482 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt"] Oct 30 05:36:07.165876 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.165839 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" path="/var/lib/kubelet/pods/7b57f9b3-b940-46df-8e3d-10509d96fab0/volumes" Oct 30 05:36:07.166620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.166600 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" path="/var/lib/kubelet/pods/cd1c7973-bf36-4bac-94de-1e67c84ad104/volumes" Oct 30 05:36:07.167057 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.167040 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:36:07.169293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.169267 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5"] Oct 30 05:36:07.205590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.205547 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:36:07.209050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.209024 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz"] Oct 30 05:36:07.247599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.247255 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:36:07.251044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:07.251016 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr"] Oct 30 05:36:08.446729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.446692 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:36:08.448378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448350 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" Oct 30 05:36:08.448546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448532 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" Oct 30 05:36:08.448686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448675 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f6e2856c-2432-472d-9930-2a7ef63e75ab" containerName="maas-api" Oct 30 05:36:08.448766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448755 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e2856c-2432-472d-9930-2a7ef63e75ab" containerName="maas-api" Oct 30 05:36:08.448860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448849 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" Oct 30 05:36:08.448939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.448928 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" Oct 30 05:36:08.449032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.449022 5058 cpu_manager.go:414] "containerMap: removing container" podUID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" containerName="maas-api" Oct 30 05:36:08.449103 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.449094 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" containerName="maas-api" Oct 30 05:36:08.449184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.449174 5058 cpu_manager.go:414] "containerMap: removing container" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" Oct 30 05:36:08.461361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461334 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" Oct 30 05:36:08.461534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461521 5058 cpu_manager.go:414] "containerMap: removing container" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" Oct 30 05:36:08.461642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461631 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" Oct 30 05:36:08.461739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461729 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" Oct 30 05:36:08.461816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461806 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" Oct 30 05:36:08.461902 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461892 5058 cpu_manager.go:414] "containerMap: removing container" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" Oct 30 05:36:08.461977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.461967 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" Oct 30 05:36:08.462051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462042 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" Oct 30 05:36:08.462130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462120 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" Oct 30 05:36:08.462246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462233 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerName="maas-api" Oct 30 05:36:08.462340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462329 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerName="maas-api" Oct 30 05:36:08.462663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462646 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e2856c-2432-472d-9930-2a7ef63e75ab" containerName="maas-api" Oct 30 05:36:08.462762 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462752 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef450b5-d521-44f0-b2b0-4808c3f503d8" containerName="payload-pre-processing" Oct 30 05:36:08.462853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462841 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabfd824-40fe-45f2-86cf-acb23b2f9d07" containerName="maas-api" Oct 30 05:36:08.462935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.462925 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="44504719-7e7d-43af-961c-f5966c48c968" containerName="istio-proxy" Oct 30 05:36:08.463020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463011 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" containerName="istio-proxy" Oct 30 05:36:08.463098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463089 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7e3e12-78ac-427c-9790-004b30634d4d" containerName="payload-processing" Oct 30 05:36:08.463168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463159 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1c7973-bf36-4bac-94de-1e67c84ad104" containerName="istio-proxy" Oct 30 05:36:08.463264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463253 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" containerName="istio-proxy" Oct 30 05:36:08.463340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463330 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a6c749-7d46-413a-a440-6d6f579f0a7e" containerName="maas-api" Oct 30 05:36:08.463419 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.463409 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b57f9b3-b940-46df-8e3d-10509d96fab0" containerName="istio-proxy" Oct 30 05:36:08.486735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.486701 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:36:08.487002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.486980 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.530236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.529350 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7226a8\"" Oct 30 05:36:08.549305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.547725 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7226a8-dockercfg-z2c8z\"" Oct 30 05:36:08.597366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.594130 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsl5\" (UniqueName: \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.597366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.594236 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.696404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.696366 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsl5\" (UniqueName: \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.696404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.696417 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.697295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.697202 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.719621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.719585 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsl5\" (UniqueName: \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:08.922201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:08.922164 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:09.163504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:09.163470 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d0cb58-f059-4b15-83bd-6e37a907234c" path="/var/lib/kubelet/pods/41d0cb58-f059-4b15-83bd-6e37a907234c/volumes" Oct 30 05:36:09.164033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:09.164019 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44504719-7e7d-43af-961c-f5966c48c968" path="/var/lib/kubelet/pods/44504719-7e7d-43af-961c-f5966c48c968/volumes" Oct 30 05:36:09.164625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:09.164608 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77" path="/var/lib/kubelet/pods/b7c5b1aa-0a5a-4663-bc8e-d2a815cdbc77/volumes" Oct 30 05:36:09.231296 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:36:09.231203 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed3c3ed_05b9_4cdb_a765_f0a30cfbdf7d.slice/crio-264d6e8e76b53f94f368184bbc5fe955af6f4494f7f52fc008ef07ab50212962 WatchSource:0}: Error finding container 264d6e8e76b53f94f368184bbc5fe955af6f4494f7f52fc008ef07ab50212962: Status 404 returned error can't find the container with id 264d6e8e76b53f94f368184bbc5fe955af6f4494f7f52fc008ef07ab50212962 Oct 30 05:36:09.232859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:09.232834 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:36:10.055085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:10.055035 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" event={"ID":"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d","Type":"ContainerStarted","Data":"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb"} Oct 30 05:36:10.055566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:10.055098 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" event={"ID":"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d","Type":"ContainerStarted","Data":"264d6e8e76b53f94f368184bbc5fe955af6f4494f7f52fc008ef07ab50212962"} Oct 30 05:36:10.055566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:10.055132 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:10.102001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:10.101946 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" podStartSLOduration=2.101930846 podStartE2EDuration="2.101930846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:10.098322808 +0000 UTC m=+3143.067817797" watchObservedRunningTime="2026-10-30 05:36:10.101930846 +0000 UTC m=+3143.071425834" Oct 30 05:36:18.834437 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:18.834393 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:36:18.835073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:18.834634 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" podUID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" containerName="payload-processing" containerID="cri-o://6845ecf491e07dabf3912fbb75b10975b6b3ca9803434e6736c63828cce0026a" gracePeriod=30 Oct 30 05:36:18.877586 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:18.877544 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:36:18.877830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:18.877799 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" podUID="9eca7964-ae42-4d06-8188-86688f231077" containerName="payload-pre-processing" containerID="cri-o://b53a7e684a549c21007cae4b03508571674e2716a1bfdbbfd353938d6dd1d9e9" gracePeriod=30 Oct 30 05:36:19.135463 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.135364 5058 generic.go:356] "Generic (PLEG): container finished" podID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" containerID="6845ecf491e07dabf3912fbb75b10975b6b3ca9803434e6736c63828cce0026a" exitCode=0 Oct 30 05:36:19.135666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.135525 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" event={"ID":"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd","Type":"ContainerDied","Data":"6845ecf491e07dabf3912fbb75b10975b6b3ca9803434e6736c63828cce0026a"} Oct 30 05:36:19.138307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.138163 5058 generic.go:356] "Generic (PLEG): container finished" podID="9eca7964-ae42-4d06-8188-86688f231077" containerID="b53a7e684a549c21007cae4b03508571674e2716a1bfdbbfd353938d6dd1d9e9" exitCode=0 Oct 30 05:36:19.138307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.138211 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" event={"ID":"9eca7964-ae42-4d06-8188-86688f231077","Type":"ContainerDied","Data":"b53a7e684a549c21007cae4b03508571674e2716a1bfdbbfd353938d6dd1d9e9"} Oct 30 05:36:19.449736 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:19.449697 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:36:19.458818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.458350 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:36:19.463588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.461823 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:36:19.524196 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524167 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\") pod \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " Oct 30 05:36:19.524414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524263 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\" (UniqueName: \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\") pod \"9eca7964-ae42-4d06-8188-86688f231077\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " Oct 30 05:36:19.524414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524330 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\" (UniqueName: \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\") pod \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\" (UID: \"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd\") " Oct 30 05:36:19.524554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524472 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\") pod \"9eca7964-ae42-4d06-8188-86688f231077\" (UID: \"9eca7964-ae42-4d06-8188-86688f231077\") " Oct 30 05:36:19.524710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524672 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume" pod "736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" (UID: "736c65c1-cc6f-4aed-92d4-0a17c53bf0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:19.524978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.524959 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.525435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.525409 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume" pod "9eca7964-ae42-4d06-8188-86688f231077" (UID: "9eca7964-ae42-4d06-8188-86688f231077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:19.527976 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.527944 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln" pod "9eca7964-ae42-4d06-8188-86688f231077" (UID: "9eca7964-ae42-4d06-8188-86688f231077"). InnerVolumeSpecName "kube-api-access-x48ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:19.528176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.528148 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z" pod "736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" (UID: "736c65c1-cc6f-4aed-92d4-0a17c53bf0cd"). InnerVolumeSpecName "kube-api-access-xfh5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:19.626634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.626594 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48ln\" (UniqueName: \"kubernetes.io/projected/9eca7964-ae42-4d06-8188-86688f231077-kube-api-access-x48ln\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.626634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.626628 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfh5z\" (UniqueName: \"kubernetes.io/projected/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd-kube-api-access-xfh5z\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.626634 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:19.626637 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eca7964-ae42-4d06-8188-86688f231077-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.147917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.147883 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" Oct 30 05:36:20.148442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.147909 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p" event={"ID":"736c65c1-cc6f-4aed-92d4-0a17c53bf0cd","Type":"ContainerDied","Data":"229cc76dc45aaa7e5a8034aa9de0a9ea0092b1589b60ce365bf80099963cc323"} Oct 30 05:36:20.148442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.147991 5058 scope.go:122] "RemoveContainer" containerID="6845ecf491e07dabf3912fbb75b10975b6b3ca9803434e6736c63828cce0026a" Oct 30 05:36:20.149618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.149590 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" Oct 30 05:36:20.149696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.149596 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk" event={"ID":"9eca7964-ae42-4d06-8188-86688f231077","Type":"ContainerDied","Data":"8f76b35adb48c65522dfb1dbf47f54f49b06e192a63f2191af2b76c6d425d24f"} Oct 30 05:36:20.162034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.162004 5058 scope.go:122] "RemoveContainer" containerID="b53a7e684a549c21007cae4b03508571674e2716a1bfdbbfd353938d6dd1d9e9" Oct 30 05:36:20.256938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.256895 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:36:20.258393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.258352 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" containerID="cri-o://5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523" gracePeriod=45 Oct 30 05:36:20.266904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.266860 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" containerID="cri-o://71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042" gracePeriod=45 Oct 30 05:36:20.279926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.279886 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p"] Oct 30 05:36:20.310943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.310910 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused" Oct 30 05:36:20.321166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.321133 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:36:20.331020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.330988 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk"] Oct 30 05:36:20.577154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.576973 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:36:20.581791 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.581764 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.645880 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.645946 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646006 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646039 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646093 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646125 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646196 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646247 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646289 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\" (UniqueName: \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646424 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\" (UniqueName: \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646461 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\") pod \"e735c001-51e4-40a2-9c01-446f39962c6f\" (UID: \"e735c001-51e4-40a2-9c01-446f39962c6f\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.646541 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\") pod \"7b1524ea-a722-4060-ad90-ebde189c382b\" (UID: \"7b1524ea-a722-4060-ad90-ebde189c382b\") " Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648041 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648286 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648468 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648645 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648693 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.650600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.648899 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651604 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651630 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651657 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651670 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651683 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.651833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.651696 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.655892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.655754 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "kube-api-access-l54vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:20.657368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.657339 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:20.658052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.658026 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:20.658636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.658579 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.658636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.658592 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm" pod "e735c001-51e4-40a2-9c01-446f39962c6f" (UID: "e735c001-51e4-40a2-9c01-446f39962c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.661171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.661138 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh" pod "7b1524ea-a722-4060-ad90-ebde189c382b" (UID: "7b1524ea-a722-4060-ad90-ebde189c382b"). InnerVolumeSpecName "kube-api-access-266hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752814 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54vx\" (UniqueName: \"kubernetes.io/projected/e735c001-51e4-40a2-9c01-446f39962c6f-kube-api-access-l54vx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752849 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266hh\" (UniqueName: \"kubernetes.io/projected/7b1524ea-a722-4060-ad90-ebde189c382b-kube-api-access-266hh\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752859 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e735c001-51e4-40a2-9c01-446f39962c6f-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752869 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e735c001-51e4-40a2-9c01-446f39962c6f-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752878 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1524ea-a722-4060-ad90-ebde189c382b-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.752900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:20.752886 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1524ea-a722-4060-ad90-ebde189c382b-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:21.070866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.070784 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:36:21.164039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.164001 5058 generic.go:356] "Generic (PLEG): container finished" podID="7b1524ea-a722-4060-ad90-ebde189c382b" containerID="71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042" exitCode=0 Oct 30 05:36:21.164579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.164100 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" Oct 30 05:36:21.167265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.167201 5058 generic.go:356] "Generic (PLEG): container finished" podID="e735c001-51e4-40a2-9c01-446f39962c6f" containerID="5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523" exitCode=0 Oct 30 05:36:21.167413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.167315 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" Oct 30 05:36:21.168774 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.168746 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" path="/var/lib/kubelet/pods/736c65c1-cc6f-4aed-92d4-0a17c53bf0cd/volumes" Oct 30 05:36:21.169408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.169386 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eca7964-ae42-4d06-8188-86688f231077" path="/var/lib/kubelet/pods/9eca7964-ae42-4d06-8188-86688f231077/volumes" Oct 30 05:36:21.170313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.170290 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" event={"ID":"7b1524ea-a722-4060-ad90-ebde189c382b","Type":"ContainerDied","Data":"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042"} Oct 30 05:36:21.170417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.170330 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm" event={"ID":"7b1524ea-a722-4060-ad90-ebde189c382b","Type":"ContainerDied","Data":"88e0a0e9f0565c941b813d6c6e1c9c19e9c54bc7725a52fe014f7d1c141b085c"} Oct 30 05:36:21.170417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.170347 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" event={"ID":"e735c001-51e4-40a2-9c01-446f39962c6f","Type":"ContainerDied","Data":"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523"} Oct 30 05:36:21.170417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.170365 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt" event={"ID":"e735c001-51e4-40a2-9c01-446f39962c6f","Type":"ContainerDied","Data":"5ba12fc08722937d8b54fc6cc4496f5c87cae9efdf33e5059fba6b982828f541"} Oct 30 05:36:21.170417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.170387 5058 scope.go:122] "RemoveContainer" containerID="71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042" Oct 30 05:36:21.187978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.187955 5058 scope.go:122] "RemoveContainer" containerID="71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042" Oct 30 05:36:21.188328 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:21.188304 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042\": container with ID starting with 71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042 not found: ID does not exist" containerID="71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042" Oct 30 05:36:21.188457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.188336 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042"} err="failed to get container status \"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042\": rpc error: code = NotFound desc = could not find container \"71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042\": container with ID starting with 71473f957d3e750fb7583b8f6c278e5d5dc12aac12b58267e572cdf8b8075042 not found: ID does not exist" Oct 30 05:36:21.188457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.188355 5058 scope.go:122] "RemoveContainer" containerID="5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523" Oct 30 05:36:21.198204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.198182 5058 scope.go:122] "RemoveContainer" containerID="5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523" Oct 30 05:36:21.198528 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:21.198504 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523\": container with ID starting with 5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523 not found: ID does not exist" containerID="5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523" Oct 30 05:36:21.198581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.198541 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523"} err="failed to get container status \"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523\": rpc error: code = NotFound desc = could not find container \"5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523\": container with ID starting with 5843f2bc6287804af410211b335a8e4ad1aa73fe56033ae94a8a085841b79523 not found: ID does not exist" Oct 30 05:36:21.240514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.240480 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 05:36:21.244197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.244167 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/premium-w4-d55037-kserve-75bb57ffb6-4m8tm"] Oct 30 05:36:21.281461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.281426 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:36:21.285393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:21.285363 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-2-w4-d55037-kserve-56bf755658-k6czt"] Oct 30 05:36:22.981436 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.981402 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:36:22.982528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982505 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" Oct 30 05:36:22.982528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982528 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982557 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982566 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982581 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9eca7964-ae42-4d06-8188-86688f231077" containerName="payload-pre-processing" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982587 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eca7964-ae42-4d06-8188-86688f231077" containerName="payload-pre-processing" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982616 5058 cpu_manager.go:414] "containerMap: removing container" podUID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" containerName="payload-processing" Oct 30 05:36:22.982690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982621 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" containerName="payload-processing" Oct 30 05:36:22.983001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982732 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" containerName="main" Oct 30 05:36:22.983001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982742 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" containerName="main" Oct 30 05:36:22.983001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982750 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eca7964-ae42-4d06-8188-86688f231077" containerName="payload-pre-processing" Oct 30 05:36:22.983001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.982760 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="736c65c1-cc6f-4aed-92d4-0a17c53bf0cd" containerName="payload-processing" Oct 30 05:36:22.988106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:22.988085 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.023372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.022652 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a62fee-openshift-default-dockercfg-d2wfr\"" Oct 30 05:36:23.033452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.033400 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:36:23.095189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095144 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095343 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095415 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095874 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095848 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095894 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095928 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.095981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.095972 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.096018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.096009 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.096063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.096050 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsrw\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197176 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197278 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197352 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197410 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197479 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsrw\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197529 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197663 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197716 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.214262 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.197763 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.251633 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.252428 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.253188 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.253994 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.255265 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.264574 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.299332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.290231 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.321236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.321185 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsrw\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.324937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.324891 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\") pod \"e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.349330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.349297 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:23.351321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.350529 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1524ea-a722-4060-ad90-ebde189c382b" path="/var/lib/kubelet/pods/7b1524ea-a722-4060-ad90-ebde189c382b/volumes" Oct 30 05:36:23.351321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.351106 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e735c001-51e4-40a2-9c01-446f39962c6f" path="/var/lib/kubelet/pods/e735c001-51e4-40a2-9c01-446f39962c6f/volumes" Oct 30 05:36:23.754328 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:36:23.754295 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1d7922_e2ef_44cb_9b00_51becde9339f.slice/crio-3520f0ca0426e4e571d9ab5e1d107d19c74b2c0036d5e9dfce405f2c1fac817a WatchSource:0}: Error finding container 3520f0ca0426e4e571d9ab5e1d107d19c74b2c0036d5e9dfce405f2c1fac817a: Status 404 returned error can't find the container with id 3520f0ca0426e4e571d9ab5e1d107d19c74b2c0036d5e9dfce405f2c1fac817a Oct 30 05:36:23.754716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.754688 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:36:23.757400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.757368 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:23.757503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.757439 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:23.757503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:23.757472 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:24.311381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:24.311340 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerStarted","Data":"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b"} Oct 30 05:36:24.311810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:24.311407 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerStarted","Data":"3520f0ca0426e4e571d9ab5e1d107d19c74b2c0036d5e9dfce405f2c1fac817a"} Oct 30 05:36:24.352679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:24.352635 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:24.360492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:24.360461 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:24.396542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:24.396487 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podStartSLOduration=2.396473435 podStartE2EDuration="2.396473435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:24.37868614 +0000 UTC m=+3157.348181129" watchObservedRunningTime="2026-10-30 05:36:24.396473435 +0000 UTC m=+3157.365968424" Oct 30 05:36:25.319648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:25.319613 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:25.320800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:25.320777 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:36:27.613912 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:27.613877 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:36:27.614446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:27.614103 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" containerID="cri-o://1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189" gracePeriod=30 Oct 30 05:36:32.513643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:32.513605 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:36:32.514069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:32.513696 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:36:32.898272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:32.898232 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:36:33.027512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027475 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027562 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027595 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027658 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027708 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027757 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.027967 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027877 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:33.028369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.027899 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.028642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.028045 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:33.028811 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.028172 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:33.028919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.028183 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:33.029013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.028668 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.029145 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.029064 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\") pod \"2340d19f-7876-4594-8113-7ae42c13e555\" (UID: \"2340d19f-7876-4594-8113-7ae42c13e555\") " Oct 30 05:36:33.029468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.029441 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:33.030027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.029998 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.030147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030031 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.030147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030046 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.030147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030059 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.030147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030073 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2340d19f-7876-4594-8113-7ae42c13e555-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.030446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030208 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "kube-api-access-4gg85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:33.030514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030440 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:33.030734 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030713 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:33.030836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.030745 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy" pod "2340d19f-7876-4594-8113-7ae42c13e555" (UID: "2340d19f-7876-4594-8113-7ae42c13e555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:33.131904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.131822 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.131904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.131853 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/2340d19f-7876-4594-8113-7ae42c13e555-kube-api-access-4gg85\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.131904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.131864 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2340d19f-7876-4594-8113-7ae42c13e555-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.131904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.131872 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2340d19f-7876-4594-8113-7ae42c13e555-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:33.408447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.408406 5058 generic.go:356] "Generic (PLEG): container finished" podID="2340d19f-7876-4594-8113-7ae42c13e555" containerID="1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189" exitCode=0 Oct 30 05:36:33.408613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.408457 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" Oct 30 05:36:33.408613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.408457 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerDied","Data":"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189"} Oct 30 05:36:33.408613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.408576 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6" event={"ID":"2340d19f-7876-4594-8113-7ae42c13e555","Type":"ContainerDied","Data":"1d8b8ea2c9e3c2782fc3f7862206b61445e19dc49b3f23c2e57319890952ce1d"} Oct 30 05:36:33.408760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.408604 5058 scope.go:122] "RemoveContainer" containerID="1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189" Oct 30 05:36:33.420120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.420098 5058 scope.go:122] "RemoveContainer" containerID="591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617" Oct 30 05:36:33.431860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.431836 5058 scope.go:122] "RemoveContainer" containerID="1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189" Oct 30 05:36:33.432371 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:33.432337 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189\": container with ID starting with 1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189 not found: ID does not exist" containerID="1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189" Oct 30 05:36:33.432461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.432386 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189"} err="failed to get container status \"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189\": rpc error: code = NotFound desc = could not find container \"1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189\": container with ID starting with 1ea5bef5969c7a72c8e972fb33064ae0df26ce42c8ee0c50530f3ebb1cf5f189 not found: ID does not exist" Oct 30 05:36:33.432461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.432416 5058 scope.go:122] "RemoveContainer" containerID="591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617" Oct 30 05:36:33.432773 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:33.432745 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617\": container with ID starting with 591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617 not found: ID does not exist" containerID="591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617" Oct 30 05:36:33.432913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.432779 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617"} err="failed to get container status \"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617\": rpc error: code = NotFound desc = could not find container \"591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617\": container with ID starting with 591ca4362344d5f18d69a29f9b0bf11e11869d68bd628c3501cb91ea9941e617 not found: ID does not exist" Oct 30 05:36:33.475870 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.475832 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:36:33.489509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:33.489465 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6"] Oct 30 05:36:35.163506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:35.163468 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2340d19f-7876-4594-8113-7ae42c13e555" path="/var/lib/kubelet/pods/2340d19f-7876-4594-8113-7ae42c13e555/volumes" Oct 30 05:36:38.955786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:38.955745 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:36:41.287768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.287726 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:36:41.288303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.287994 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" podUID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" containerName="maas-api" containerID="cri-o://332cb9f2dc7ee0c91a7853c2c9ab17f7002ca0a9e0398cd838ec8fb77f8c49da" gracePeriod=30 Oct 30 05:36:41.552450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.551594 5058 generic.go:356] "Generic (PLEG): container finished" podID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" containerID="332cb9f2dc7ee0c91a7853c2c9ab17f7002ca0a9e0398cd838ec8fb77f8c49da" exitCode=0 Oct 30 05:36:41.552450 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.551726 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" event={"ID":"bbe7d883-0a03-41ad-ad68-1698f4a650b6","Type":"ContainerDied","Data":"332cb9f2dc7ee0c91a7853c2c9ab17f7002ca0a9e0398cd838ec8fb77f8c49da"} Oct 30 05:36:41.667816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.667786 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:36:41.805818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.805635 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\") pod \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " Oct 30 05:36:41.805818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.805757 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\" (UniqueName: \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\") pod \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " Oct 30 05:36:41.806069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.805837 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\") pod \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " Oct 30 05:36:41.806069 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.805863 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\") pod \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\" (UID: \"bbe7d883-0a03-41ad-ad68-1698f4a650b6\") " Oct 30 05:36:41.806664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.806631 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca" pod "bbe7d883-0a03-41ad-ad68-1698f4a650b6" (UID: "bbe7d883-0a03-41ad-ad68-1698f4a650b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:41.808749 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.808715 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv" pod "bbe7d883-0a03-41ad-ad68-1698f4a650b6" (UID: "bbe7d883-0a03-41ad-ad68-1698f4a650b6"). InnerVolumeSpecName "kube-api-access-s9rxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:41.809096 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.809073 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs" pod "bbe7d883-0a03-41ad-ad68-1698f4a650b6" (UID: "bbe7d883-0a03-41ad-ad68-1698f4a650b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:41.809186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.809101 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls" pod "bbe7d883-0a03-41ad-ad68-1698f4a650b6" (UID: "bbe7d883-0a03-41ad-ad68-1698f4a650b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:41.948243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.916645 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:41.948513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.948484 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rxv\" (UniqueName: \"kubernetes.io/projected/bbe7d883-0a03-41ad-ad68-1698f4a650b6-kube-api-access-s9rxv\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:41.974376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.974335 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7d883-0a03-41ad-ad68-1698f4a650b6-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:41.974721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:41.974696 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7d883-0a03-41ad-ad68-1698f4a650b6-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:42.586244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:42.584380 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" Oct 30 05:36:42.586244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:42.585871 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp" event={"ID":"bbe7d883-0a03-41ad-ad68-1698f4a650b6","Type":"ContainerDied","Data":"1cdb5390f035dc8c31a04e82a36c4baca97f51bc34684aba0b2d32d438cd9e7c"} Oct 30 05:36:42.586244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:42.585939 5058 scope.go:122] "RemoveContainer" containerID="332cb9f2dc7ee0c91a7853c2c9ab17f7002ca0a9e0398cd838ec8fb77f8c49da" Oct 30 05:36:42.704979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:42.704947 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:36:42.717497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:42.717459 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02cffa-69d567c9cf-s95tp"] Oct 30 05:36:43.177360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:43.177327 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" path="/var/lib/kubelet/pods/bbe7d883-0a03-41ad-ad68-1698f4a650b6/volumes" Oct 30 05:36:43.956280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:43.956235 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" containerID="cri-o://f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e" gracePeriod=25 Oct 30 05:36:47.294752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.294675 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:47.294752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.294727 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:47.632779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.632736 5058 generic.go:356] "Generic (PLEG): container finished" podID="61810822-ab44-48db-8060-240e7c848ca0" containerID="1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200" exitCode=0 Oct 30 05:36:47.632779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.632760 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerDied","Data":"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200"} Oct 30 05:36:47.633755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.633734 5058 scope.go:122] "RemoveContainer" containerID="1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200" Oct 30 05:36:47.635458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.635429 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:47.635554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.635497 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:47.635554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:47.635529 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:36:48.658088 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:48.658051 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerStarted","Data":"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919"} Oct 30 05:36:49.192285 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:36:49.192246 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:36:49.609259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:49.609156 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:36:49.609566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:49.609361 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:49.609566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:49.609434 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:50.594766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.594691 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:36:50.595695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.595384 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" containerID="cri-o://197d25f19c8eafa2c55748f7ef5736e7ad5739328c36d8f9b68d670c17d23b17" gracePeriod=30 Oct 30 05:36:50.610122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.610093 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:36:50.610351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.610326 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:50.610478 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.610452 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:50.610964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.610910 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" containerID="cri-o://51ebefcfacc2e155394bc4ebba9fad21028cd6939c1bd5abac42d0d69241637d" gracePeriod=30 Oct 30 05:36:50.615746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.615721 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:36:50.615981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.615945 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" containerID="cri-o://ca2d42f490aa67a5ef63e0a22bde8334871805d887d12a92f9a44d7bb9fbbcca" gracePeriod=30 Oct 30 05:36:50.622122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.622094 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:36:50.622295 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.622275 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" containerID="cri-o://0ec075d6944e9c491283b76001ed9505f11d5343b2db71006080d79ebb89ffda" gracePeriod=30 Oct 30 05:36:50.950459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.950408 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:36:50.950637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:50.950475 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:36:51.555299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:51.555254 5058 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:36:51.555502 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:51.555328 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:36:51.609637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:51.609599 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:51.610049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:51.609666 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:52.610326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:52.610291 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:52.610806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:52.610353 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:53.400181 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:53.400139 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:36:53.609420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:53.609373 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:36:53.609611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:53.609438 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 05:36:54.615356 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:54.615322 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:36:54.615818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:54.615796 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:36:54.616657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:54.616636 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:36:55.833772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.833730 5058 generic.go:356] "Generic (PLEG): container finished" podID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerID="0ec075d6944e9c491283b76001ed9505f11d5343b2db71006080d79ebb89ffda" exitCode=0 Oct 30 05:36:55.834173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.833800 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" event={"ID":"5eb5320a-45ef-4161-92a2-c9dabe83bd27","Type":"ContainerDied","Data":"0ec075d6944e9c491283b76001ed9505f11d5343b2db71006080d79ebb89ffda"} Oct 30 05:36:55.837619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.835918 5058 generic.go:356] "Generic (PLEG): container finished" podID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerID="51ebefcfacc2e155394bc4ebba9fad21028cd6939c1bd5abac42d0d69241637d" exitCode=0 Oct 30 05:36:55.837619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.836093 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" event={"ID":"03092fc6-badb-4e1c-9114-82b110f0eec2","Type":"ContainerDied","Data":"51ebefcfacc2e155394bc4ebba9fad21028cd6939c1bd5abac42d0d69241637d"} Oct 30 05:36:55.845480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.845432 5058 generic.go:356] "Generic (PLEG): container finished" podID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerID="197d25f19c8eafa2c55748f7ef5736e7ad5739328c36d8f9b68d670c17d23b17" exitCode=0 Oct 30 05:36:55.845650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.845583 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" event={"ID":"06864d1a-d062-4b6c-ad35-90c40a1730ed","Type":"ContainerDied","Data":"197d25f19c8eafa2c55748f7ef5736e7ad5739328c36d8f9b68d670c17d23b17"} Oct 30 05:36:55.848345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.848245 5058 generic.go:356] "Generic (PLEG): container finished" podID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerID="ca2d42f490aa67a5ef63e0a22bde8334871805d887d12a92f9a44d7bb9fbbcca" exitCode=0 Oct 30 05:36:55.856053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.856009 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" event={"ID":"36d4ed5d-1090-44be-af1a-33db8320a84b","Type":"ContainerDied","Data":"ca2d42f490aa67a5ef63e0a22bde8334871805d887d12a92f9a44d7bb9fbbcca"} Oct 30 05:36:55.968670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:55.968635 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:36:56.004659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.004623 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.038793 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.038846 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.038882 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.038954 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039001 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039029 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039087 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039119 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039202 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039301 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039354 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039426 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039504 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039535 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039577 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\") pod \"36d4ed5d-1090-44be-af1a-33db8320a84b\" (UID: \"36d4ed5d-1090-44be-af1a-33db8320a84b\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039628 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039678 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.043473 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.039715 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\") pod \"06864d1a-d062-4b6c-ad35-90c40a1730ed\" (UID: \"06864d1a-d062-4b6c-ad35-90c40a1730ed\") " Oct 30 05:36:56.045952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.044790 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.048502 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.048969 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.049209 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.050115 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.051868 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:56.052183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.052171 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.052758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.052315 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.053055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.053027 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.053233 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.053198 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:56.054911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.054718 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:56.055395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.055367 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:56.055497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.055478 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.056698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.056674 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.056801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.056766 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "kube-api-access-8xqpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.056871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.056833 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.058561 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.058526 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt" pod "36d4ed5d-1090-44be-af1a-33db8320a84b" (UID: "36d4ed5d-1090-44be-af1a-33db8320a84b"). InnerVolumeSpecName "kube-api-access-m86zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.060266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.060230 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token" pod "06864d1a-d062-4b6c-ad35-90c40a1730ed" (UID: "06864d1a-d062-4b6c-ad35-90c40a1730ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.133990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.133959 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:36:56.136268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.136205 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:36:56.143074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143033 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143075 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143090 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143103 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143116 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06864d1a-d062-4b6c-ad35-90c40a1730ed-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143129 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143141 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143155 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143168 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06864d1a-d062-4b6c-ad35-90c40a1730ed-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143181 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143194 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143209 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143240 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86zt\" (UniqueName: \"kubernetes.io/projected/36d4ed5d-1090-44be-af1a-33db8320a84b-kube-api-access-m86zt\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143254 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqpj\" (UniqueName: \"kubernetes.io/projected/06864d1a-d062-4b6c-ad35-90c40a1730ed-kube-api-access-8xqpj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143268 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4ed5d-1090-44be-af1a-33db8320a84b-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143281 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143295 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4ed5d-1090-44be-af1a-33db8320a84b-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.143313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.143307 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06864d1a-d062-4b6c-ad35-90c40a1730ed-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.244855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.244743 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.244881 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.244929 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.244961 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245003 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245050 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245092 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245130 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245167 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245202 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245334 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245371 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245406 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245552 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245576 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245614 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\") pod \"03092fc6-badb-4e1c-9114-82b110f0eec2\" (UID: \"03092fc6-badb-4e1c-9114-82b110f0eec2\") " Oct 30 05:36:56.245760 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245673 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.245819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.245763 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\") pod \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\" (UID: \"5eb5320a-45ef-4161-92a2-c9dabe83bd27\") " Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.246190 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.246483 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.246682 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.246695 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.246717 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.247051 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.247256 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.247553 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.247938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.247813 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:56.248518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.248240 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:56.249373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.249342 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.249539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.249405 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.253047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.253015 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:56.253237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.253177 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.253536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.253513 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.253677 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.253660 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "kube-api-access-2nm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.253758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.253687 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "kube-api-access-nspjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.254281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.254259 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:56.254384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.254321 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token" pod "03092fc6-badb-4e1c-9114-82b110f0eec2" (UID: "03092fc6-badb-4e1c-9114-82b110f0eec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.255393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.255365 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token" pod "5eb5320a-45ef-4161-92a2-c9dabe83bd27" (UID: "5eb5320a-45ef-4161-92a2-c9dabe83bd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.347909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347869 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03092fc6-badb-4e1c-9114-82b110f0eec2-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.347909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347903 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.347909 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347913 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347923 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347932 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347940 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347949 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347958 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347968 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspjm\" (UniqueName: \"kubernetes.io/projected/03092fc6-badb-4e1c-9114-82b110f0eec2-kube-api-access-nspjm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347977 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.347995 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.348003 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.348012 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03092fc6-badb-4e1c-9114-82b110f0eec2-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.348019 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.348026 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb5320a-45ef-4161-92a2-c9dabe83bd27-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.348171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.348036 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nm47\" (UniqueName: \"kubernetes.io/projected/5eb5320a-45ef-4161-92a2-c9dabe83bd27-kube-api-access-2nm47\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.857128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.857094 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" Oct 30 05:36:56.857626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.857124 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49" event={"ID":"03092fc6-badb-4e1c-9114-82b110f0eec2","Type":"ContainerDied","Data":"962cd3b8532e92c94f1a7117d0b3c950c8674398a659623991a31eb3f9e67f45"} Oct 30 05:36:56.857626 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.857193 5058 scope.go:122] "RemoveContainer" containerID="51ebefcfacc2e155394bc4ebba9fad21028cd6939c1bd5abac42d0d69241637d" Oct 30 05:36:56.859140 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.859117 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" Oct 30 05:36:56.859261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.859145 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v" event={"ID":"06864d1a-d062-4b6c-ad35-90c40a1730ed","Type":"ContainerDied","Data":"1b58b2e4ddd19c3b750c89cc2bd1ec71c69d28c0c54e48de71cb9b5a02877c1c"} Oct 30 05:36:56.860855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.860828 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" Oct 30 05:36:56.860995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.860849 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2" event={"ID":"36d4ed5d-1090-44be-af1a-33db8320a84b","Type":"ContainerDied","Data":"d90d3d20744fe558d7e8a7473170abb85d8428b70a6665228efa109865fd4ae7"} Oct 30 05:36:56.862676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.862635 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" event={"ID":"5eb5320a-45ef-4161-92a2-c9dabe83bd27","Type":"ContainerDied","Data":"a78f040c134c2c45c8b1b9ab0fc472b7faf658a8c042a711ee20f49f852a6bc3"} Oct 30 05:36:56.862802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.862647 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x" Oct 30 05:36:56.871516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.871493 5058 scope.go:122] "RemoveContainer" containerID="197d25f19c8eafa2c55748f7ef5736e7ad5739328c36d8f9b68d670c17d23b17" Oct 30 05:36:56.897607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.897580 5058 scope.go:122] "RemoveContainer" containerID="ca2d42f490aa67a5ef63e0a22bde8334871805d887d12a92f9a44d7bb9fbbcca" Oct 30 05:36:56.908002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.907975 5058 scope.go:122] "RemoveContainer" containerID="0ec075d6944e9c491283b76001ed9505f11d5343b2db71006080d79ebb89ffda" Oct 30 05:36:56.916801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.916773 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:36:56.930013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.929985 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v"] Oct 30 05:36:56.975901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.975869 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:36:56.981399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:56.981371 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49"] Oct 30 05:36:57.036042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.036009 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:36:57.044187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.044154 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2"] Oct 30 05:36:57.081588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.081554 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:36:57.087530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.087498 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x"] Oct 30 05:36:57.164499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.164467 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" path="/var/lib/kubelet/pods/03092fc6-badb-4e1c-9114-82b110f0eec2/volumes" Oct 30 05:36:57.165011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.164995 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" path="/var/lib/kubelet/pods/06864d1a-d062-4b6c-ad35-90c40a1730ed/volumes" Oct 30 05:36:57.165486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.165471 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" path="/var/lib/kubelet/pods/36d4ed5d-1090-44be-af1a-33db8320a84b/volumes" Oct 30 05:36:57.166246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:36:57.166212 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" path="/var/lib/kubelet/pods/5eb5320a-45ef-4161-92a2-c9dabe83bd27/volumes" Oct 30 05:37:01.089135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.089105 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 05:37:01.090087 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.089328 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" podUID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" containerName="maas-api" containerID="cri-o://7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9" gracePeriod=30 Oct 30 05:37:01.422594 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.422558 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 05:37:01.522654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.522612 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\" (UniqueName: \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\") pod \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " Oct 30 05:37:01.522921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.522672 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\") pod \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " Oct 30 05:37:01.522921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.522761 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") pod \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " Oct 30 05:37:01.522921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.522867 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\") pod \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\" (UID: \"3bf3ed89-c26a-43c2-8ec7-7db1888117d4\") " Oct 30 05:37:01.525054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.523935 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca" pod "3bf3ed89-c26a-43c2-8ec7-7db1888117d4" (UID: "3bf3ed89-c26a-43c2-8ec7-7db1888117d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:01.536821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.535845 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5" pod "3bf3ed89-c26a-43c2-8ec7-7db1888117d4" (UID: "3bf3ed89-c26a-43c2-8ec7-7db1888117d4"). InnerVolumeSpecName "kube-api-access-z84w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:01.538355 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.538271 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs" pod "3bf3ed89-c26a-43c2-8ec7-7db1888117d4" (UID: "3bf3ed89-c26a-43c2-8ec7-7db1888117d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:01.540358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.540328 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls" pod "3bf3ed89-c26a-43c2-8ec7-7db1888117d4" (UID: "3bf3ed89-c26a-43c2-8ec7-7db1888117d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:01.625260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.625158 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:01.625260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.625193 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z84w5\" (UniqueName: \"kubernetes.io/projected/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-kube-api-access-z84w5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:01.625260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.625202 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:01.625260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.625212 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ed89-c26a-43c2-8ec7-7db1888117d4-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:01.907000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.906959 5058 generic.go:356] "Generic (PLEG): container finished" podID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" containerID="7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9" exitCode=0 Oct 30 05:37:01.907206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.907165 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" event={"ID":"3bf3ed89-c26a-43c2-8ec7-7db1888117d4","Type":"ContainerDied","Data":"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9"} Oct 30 05:37:01.907308 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.907292 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" event={"ID":"3bf3ed89-c26a-43c2-8ec7-7db1888117d4","Type":"ContainerDied","Data":"92971e008d5c28ac0a85c37a933f1943427d424f769e7371926e10f6c65c81e3"} Oct 30 05:37:01.907384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.907328 5058 scope.go:122] "RemoveContainer" containerID="7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9" Oct 30 05:37:01.907496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.907483 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9" Oct 30 05:37:01.929518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.924739 5058 scope.go:122] "RemoveContainer" containerID="7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9" Oct 30 05:37:01.929518 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:01.928514 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9\": container with ID starting with 7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9 not found: ID does not exist" containerID="7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9" Oct 30 05:37:01.929518 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.928556 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9"} err="failed to get container status \"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9\": rpc error: code = NotFound desc = could not find container \"7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9\": container with ID starting with 7db0cbc12a3963d85311b1acaf0849ac3e1e99cddce0ee5cd4c25fad27bd90f9 not found: ID does not exist" Oct 30 05:37:01.980513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.980465 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 05:37:01.994660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:01.994625 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d55037-589d6646bd-qvkl9"] Oct 30 05:37:02.370545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.370459 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:37:02.371384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371362 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371384 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371406 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" containerName="maas-api" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371413 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" containerName="maas-api" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371437 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371444 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371456 5058 cpu_manager.go:414] "containerMap: removing container" podUID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" containerName="maas-api" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371463 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" containerName="maas-api" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371483 5058 cpu_manager.go:414] "containerMap: removing container" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" Oct 30 05:37:02.371509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371490 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371531 5058 cpu_manager.go:414] "containerMap: removing container" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371541 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371572 5058 cpu_manager.go:414] "containerMap: removing container" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371581 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371758 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="03092fc6-badb-4e1c-9114-82b110f0eec2" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371776 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371787 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371796 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" containerName="maas-api" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371807 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb5320a-45ef-4161-92a2-c9dabe83bd27" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371816 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe7d883-0a03-41ad-ad68-1698f4a650b6" containerName="maas-api" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371825 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="06864d1a-d062-4b6c-ad35-90c40a1730ed" containerName="istio-proxy" Oct 30 05:37:02.371973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.371838 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d4ed5d-1090-44be-af1a-33db8320a84b" containerName="istio-proxy" Oct 30 05:37:02.377553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.377522 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.394941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.394754 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a62fee-serving-cert\"" Oct 30 05:37:02.397740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.397710 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:37:02.442910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.442877 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.443098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.442926 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qndj\" (UniqueName: \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.443098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.443049 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.443281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.443248 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.545115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.545067 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.545397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.545199 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.545397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.545274 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qndj\" (UniqueName: \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.545397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.545328 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.545977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.545955 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.547699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.547678 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.547838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.547816 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.566918 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.566884 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qndj\" (UniqueName: \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\") pod \"maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:02.693763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:02.693720 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:03.000068 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:37:03.000031 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbda810_6e57_412a_a569_fbfb50859a32.slice/crio-4d0e513e7c77350a883741d9de5e790ba72947633825f62d5800c538203662e3 WatchSource:0}: Error finding container 4d0e513e7c77350a883741d9de5e790ba72947633825f62d5800c538203662e3: Status 404 returned error can't find the container with id 4d0e513e7c77350a883741d9de5e790ba72947633825f62d5800c538203662e3 Oct 30 05:37:03.000830 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.000805 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:37:03.163313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.163274 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf3ed89-c26a-43c2-8ec7-7db1888117d4" path="/var/lib/kubelet/pods/3bf3ed89-c26a-43c2-8ec7-7db1888117d4/volumes" Oct 30 05:37:03.400434 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.400402 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:37:03.926062 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.926023 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" event={"ID":"5cbda810-6e57-412a-a569-fbfb50859a32","Type":"ContainerStarted","Data":"266e85bab3524aa9e7112b52a4f0bf00c617cf84c09a07ea2d0f43a83db416a8"} Oct 30 05:37:03.926244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.926081 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" event={"ID":"5cbda810-6e57-412a-a569-fbfb50859a32","Type":"ContainerStarted","Data":"4d0e513e7c77350a883741d9de5e790ba72947633825f62d5800c538203662e3"} Oct 30 05:37:03.926244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.926110 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:03.986250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:03.985065 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" podStartSLOduration=1.639154468 podStartE2EDuration="1.985017664s" totalImagesPullingTime="345.863196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:03.976528813 +0000 UTC m=+3196.946023804" watchObservedRunningTime="2026-10-30 05:37:03.985017664 +0000 UTC m=+3196.954512654" Oct 30 05:37:04.299630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.299597 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:37:04.381315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.381280 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\" (UniqueName: \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\") pod \"06971047-7680-406d-84a7-02f31bd1a2d9\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " Oct 30 05:37:04.381536 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.381356 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\") pod \"06971047-7680-406d-84a7-02f31bd1a2d9\" (UID: \"06971047-7680-406d-84a7-02f31bd1a2d9\") " Oct 30 05:37:04.382373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.382344 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume" pod "06971047-7680-406d-84a7-02f31bd1a2d9" (UID: "06971047-7680-406d-84a7-02f31bd1a2d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:04.383794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.383761 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72" pod "06971047-7680-406d-84a7-02f31bd1a2d9" (UID: "06971047-7680-406d-84a7-02f31bd1a2d9"). InnerVolumeSpecName "kube-api-access-tjj72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:04.483358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.483263 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjj72\" (UniqueName: \"kubernetes.io/projected/06971047-7680-406d-84a7-02f31bd1a2d9-kube-api-access-tjj72\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:04.483358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.483304 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971047-7680-406d-84a7-02f31bd1a2d9-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:04.934638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.934600 5058 generic.go:356] "Generic (PLEG): container finished" podID="06971047-7680-406d-84a7-02f31bd1a2d9" containerID="f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e" exitCode=0 Oct 30 05:37:04.934860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.934665 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" Oct 30 05:37:04.934860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.934670 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" event={"ID":"06971047-7680-406d-84a7-02f31bd1a2d9","Type":"ContainerDied","Data":"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e"} Oct 30 05:37:04.934860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.934770 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww" event={"ID":"06971047-7680-406d-84a7-02f31bd1a2d9","Type":"ContainerDied","Data":"14d571578e88103031ed185e5bcd2b7864882c94dd236bb624df6b7d462adf54"} Oct 30 05:37:04.934860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.934802 5058 scope.go:122] "RemoveContainer" containerID="f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e" Oct 30 05:37:04.946092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.946068 5058 scope.go:122] "RemoveContainer" containerID="f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e" Oct 30 05:37:04.946473 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:04.946450 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e\": container with ID starting with f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e not found: ID does not exist" containerID="f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e" Oct 30 05:37:04.946554 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.946479 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e"} err="failed to get container status \"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e\": rpc error: code = NotFound desc = could not find container \"f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e\": container with ID starting with f2d4629e9a4c67547e29cf7576afb024a4e8e669a38bcc6b8bc3eaec0e072f1e not found: ID does not exist" Oct 30 05:37:04.991025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:04.990980 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:37:05.004370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:05.004335 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww"] Oct 30 05:37:05.162799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:05.162764 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" path="/var/lib/kubelet/pods/06971047-7680-406d-84a7-02f31bd1a2d9/volumes" Oct 30 05:37:09.940304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:09.940266 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:37:12.475123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.475082 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:37:12.475675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.475340 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" podUID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" containerName="maas-api" containerID="cri-o://83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49" gracePeriod=30 Oct 30 05:37:12.760675 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.760631 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:37:12.847875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.847830 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\") pod \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " Oct 30 05:37:12.848105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.848043 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\") pod \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " Oct 30 05:37:12.848187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.848134 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\" (UniqueName: \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\") pod \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " Oct 30 05:37:12.848340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.848210 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\") pod \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\" (UID: \"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6\") " Oct 30 05:37:12.849051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.848988 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca" pod "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" (UID: "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:12.852171 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.852140 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls" pod "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" (UID: "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:12.852349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.852311 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs" pod "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" (UID: "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:12.860273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.859623 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw" pod "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" (UID: "7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6"). InnerVolumeSpecName "kube-api-access-7ljbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:12.950663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.950616 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:12.950663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.950654 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljbw\" (UniqueName: \"kubernetes.io/projected/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-kube-api-access-7ljbw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:12.950663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.950668 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:12.950924 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:12.950681 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.002743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.002651 5058 generic.go:356] "Generic (PLEG): container finished" podID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" containerID="83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49" exitCode=0 Oct 30 05:37:13.002743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.002712 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" Oct 30 05:37:13.002960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.002736 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" event={"ID":"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6","Type":"ContainerDied","Data":"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49"} Oct 30 05:37:13.002960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.002804 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs" event={"ID":"7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6","Type":"ContainerDied","Data":"e7a87fe7ff266a6f9efd12cbd9e5887241b3b6b043f90da31164c868f0231c11"} Oct 30 05:37:13.002960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.002827 5058 scope.go:122] "RemoveContainer" containerID="83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49" Oct 30 05:37:13.014206 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.014178 5058 scope.go:122] "RemoveContainer" containerID="83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49" Oct 30 05:37:13.014553 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:13.014533 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49\": container with ID starting with 83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49 not found: ID does not exist" containerID="83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49" Oct 30 05:37:13.014633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.014560 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49"} err="failed to get container status \"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49\": rpc error: code = NotFound desc = could not find container \"83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49\": container with ID starting with 83084509bf23f14b3e1efefedfb9aac059c2582c56e83c916eb4fa67f3e0ab49 not found: ID does not exist" Oct 30 05:37:13.059624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.059592 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:37:13.063956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.063922 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4bc875-74f5859967-j4rjs"] Oct 30 05:37:13.352100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:13.350976 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" path="/var/lib/kubelet/pods/7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6/volumes" Oct 30 05:37:19.191886 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:19.191843 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:37:19.550986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.550901 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:37:19.551600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551584 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" containerName="maas-api" Oct 30 05:37:19.551600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551600 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" containerName="maas-api" Oct 30 05:37:19.551700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551625 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:19.551700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551630 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2340d19f-7876-4594-8113-7ae42c13e555" containerName="istio-proxy" Oct 30 05:37:19.551700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551657 5058 cpu_manager.go:414] "containerMap: removing container" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" Oct 30 05:37:19.551700 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551663 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" Oct 30 05:37:19.551838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551767 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="06971047-7680-406d-84a7-02f31bd1a2d9" containerName="payload-processing" Oct 30 05:37:19.551838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.551780 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1e3ee8-1dc8-410b-8eb9-e7a15a6888d6" containerName="maas-api" Oct 30 05:37:19.555372 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.555343 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.573077 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.573049 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:37:19.588589 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.588556 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7a4600-dockercfg-rjz56\"" Oct 30 05:37:19.588765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.588690 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7a4600\"" Oct 30 05:37:19.678041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.677970 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpn7j\" (UniqueName: \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.678508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.678464 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.753240 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.751322 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:37:19.784364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.784236 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpn7j\" (UniqueName: \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.784800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.784761 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.786013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.785986 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.799243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.798823 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:37:19.799243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.799003 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:19.817368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.817169 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpn7j\" (UniqueName: \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.869415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.869384 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:19.886109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.886080 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kj2\" (UniqueName: \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:19.886278 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.886129 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:19.988014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.987967 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kj2\" (UniqueName: \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:19.988200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.988054 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:19.989358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:19.989326 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:20.014191 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.014159 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kj2\" (UniqueName: \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\") pod \"payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:20.169330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.169284 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:20.186376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.183442 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:37:20.186376 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:37:20.183948 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ad3d95_f894_40d1_a077_488e4250b79c.slice/crio-eb714a341db7020fca04187a0a75d756efbec204cd4c796930bfcaa632474913 WatchSource:0}: Error finding container eb714a341db7020fca04187a0a75d756efbec204cd4c796930bfcaa632474913: Status 404 returned error can't find the container with id eb714a341db7020fca04187a0a75d756efbec204cd4c796930bfcaa632474913 Oct 30 05:37:20.488466 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.486694 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:37:20.498282 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:37:20.495290 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5308d7_6e00_4944_ab33_00c1a59a27b2.slice/crio-4a5e02236da3abfd3704339084e1daa327a7ffaff1f3346e670c233ae8609ff3 WatchSource:0}: Error finding container 4a5e02236da3abfd3704339084e1daa327a7ffaff1f3346e670c233ae8609ff3: Status 404 returned error can't find the container with id 4a5e02236da3abfd3704339084e1daa327a7ffaff1f3346e670c233ae8609ff3 Oct 30 05:37:20.759582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.759492 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:37:20.760371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:20.760014 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" containerID="cri-o://35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001" gracePeriod=30 Oct 30 05:37:21.071526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.071426 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" event={"ID":"3e5308d7-6e00-4944-ab33-00c1a59a27b2","Type":"ContainerStarted","Data":"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede"} Oct 30 05:37:21.071526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.071494 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" event={"ID":"3e5308d7-6e00-4944-ab33-00c1a59a27b2","Type":"ContainerStarted","Data":"4a5e02236da3abfd3704339084e1daa327a7ffaff1f3346e670c233ae8609ff3"} Oct 30 05:37:21.071526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.071522 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:21.073620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.073590 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" event={"ID":"36ad3d95-f894-40d1-a077-488e4250b79c","Type":"ContainerStarted","Data":"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9"} Oct 30 05:37:21.073767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.073636 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" event={"ID":"36ad3d95-f894-40d1-a077-488e4250b79c","Type":"ContainerStarted","Data":"eb714a341db7020fca04187a0a75d756efbec204cd4c796930bfcaa632474913"} Oct 30 05:37:21.073767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.073696 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:21.121612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.121561 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" podStartSLOduration=2.12154601 podStartE2EDuration="2.12154601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:21.117477109 +0000 UTC m=+3214.086972088" watchObservedRunningTime="2026-10-30 05:37:21.12154601 +0000 UTC m=+3214.091041000" Oct 30 05:37:21.162726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.162678 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:37:21.172638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.172575 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" podStartSLOduration=2.172562357 podStartE2EDuration="2.172562357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:21.169208109 +0000 UTC m=+3214.138703098" watchObservedRunningTime="2026-10-30 05:37:21.172562357 +0000 UTC m=+3214.142057345" Oct 30 05:37:21.205864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:21.205826 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:37:25.988688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:25.988660 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:37:26.086772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.086681 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.086772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.086735 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.086860 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.086962 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.086996 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087041 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087440 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087550 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087350 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:26.087617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087586 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087630 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\") pod \"c43bbdb4-3917-460e-8c18-d81926232c43\" (UID: \"c43bbdb4-3917-460e-8c18-d81926232c43\") " Oct 30 05:37:26.087738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087708 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:26.087738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087480 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:26.087932 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.087914 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:26.088052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.088026 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:26.088913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.088884 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.089044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.089026 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.089201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.089184 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.089493 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.089467 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.089599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.089586 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43bbdb4-3917-460e-8c18-d81926232c43-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.089802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.089775 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:26.090501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.090468 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:37:26.090629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.090604 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "kube-api-access-7xpbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:26.091258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.091183 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token" pod "c43bbdb4-3917-460e-8c18-d81926232c43" (UID: "c43bbdb4-3917-460e-8c18-d81926232c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:26.163737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.163695 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" containerID="cri-o://45aff2d006b7503ef4765a8b4af5fdacd42070e0db9d0a2424b161cae7d96c05" gracePeriod=25 Oct 30 05:37:26.191067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.191024 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpbg\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-kube-api-access-7xpbg\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.191067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.191063 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43bbdb4-3917-460e-8c18-d81926232c43-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.191067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.191073 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43bbdb4-3917-460e-8c18-d81926232c43-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.191331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.191083 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43bbdb4-3917-460e-8c18-d81926232c43-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:26.206139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.206095 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" containerID="cri-o://32d7486602eb13f84fe0055143cb9c1e6eb7312d2586d0285401436560e91ac3" gracePeriod=25 Oct 30 05:37:26.207371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.207346 5058 generic.go:356] "Generic (PLEG): container finished" podID="c43bbdb4-3917-460e-8c18-d81926232c43" containerID="35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001" exitCode=0 Oct 30 05:37:26.207490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.207417 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" Oct 30 05:37:26.207490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.207442 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerDied","Data":"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001"} Oct 30 05:37:26.207611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.207518 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg" event={"ID":"c43bbdb4-3917-460e-8c18-d81926232c43","Type":"ContainerDied","Data":"0b73110aee0c1c16c78f45072d6ac83b30e72a01061bacea632ec5ad841f74af"} Oct 30 05:37:26.207611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.207552 5058 scope.go:122] "RemoveContainer" containerID="35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001" Oct 30 05:37:26.218786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.218767 5058 scope.go:122] "RemoveContainer" containerID="d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f" Oct 30 05:37:26.256151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.256128 5058 scope.go:122] "RemoveContainer" containerID="35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001" Oct 30 05:37:26.256551 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:26.256523 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001\": container with ID starting with 35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001 not found: ID does not exist" containerID="35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001" Oct 30 05:37:26.256674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.256559 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001"} err="failed to get container status \"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001\": rpc error: code = NotFound desc = could not find container \"35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001\": container with ID starting with 35c8d88ba9dc0d8df278aa6796193e67414d24beeec2bdd5a474a67f1a494001 not found: ID does not exist" Oct 30 05:37:26.256674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.256580 5058 scope.go:122] "RemoveContainer" containerID="d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f" Oct 30 05:37:26.256905 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:26.256884 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f\": container with ID starting with d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f not found: ID does not exist" containerID="d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f" Oct 30 05:37:26.256978 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.256909 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f"} err="failed to get container status \"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f\": rpc error: code = NotFound desc = could not find container \"d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f\": container with ID starting with d91ff890207eb0acb7f9828713819e19fcf33452f6b48a094d3dcaeec192d08f not found: ID does not exist" Oct 30 05:37:26.276768 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.276733 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:37:26.280825 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:26.280801 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg"] Oct 30 05:37:27.162959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:27.162915 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" path="/var/lib/kubelet/pods/c43bbdb4-3917-460e-8c18-d81926232c43/volumes" Oct 30 05:37:32.083028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:32.082995 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:37:32.083499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:32.083079 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:37:34.296033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.295992 5058 generic.go:356] "Generic (PLEG): container finished" podID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerID="d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b" exitCode=0 Oct 30 05:37:34.296611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.296068 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerDied","Data":"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b"} Oct 30 05:37:34.296611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.296578 5058 scope.go:122] "RemoveContainer" containerID="d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b" Oct 30 05:37:34.298255 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.298208 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:37:34.298338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.298296 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:37:34.298338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.298331 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:37:34.515477 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.515437 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:37:34.520363 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:34.520323 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:37:35.305411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:35.305376 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerStarted","Data":"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39"} Oct 30 05:37:35.352689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:35.352646 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:37:35.352955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:35.352924 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:35.353034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:35.353009 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:36.350568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:36.350533 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l_3e5308d7-6e00-4944-ab33-00c1a59a27b2/payload-processing/0.log" Oct 30 05:37:36.351979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:36.351945 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:36.352199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:36.352169 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:36.793022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:36.792982 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:37:36.793275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:36.793247 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" containerID="cri-o://930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb" gracePeriod=30 Oct 30 05:37:37.086693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:37.086609 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:37:37.086693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:37.086673 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:37:37.352177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:37.352078 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:37.352177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:37.352135 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:38.367246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:38.366378 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:38.367246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:38.366439 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:39.352657 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:39.352623 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:39.352901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:39.352683 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:40.320371 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:40.320331 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:37:40.352409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:40.352366 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:37:40.352632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:40.352433 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 05:37:41.357612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:41.357581 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:37:41.358790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:41.358764 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:37:42.068777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.068704 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:37:42.125996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.125923 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.125996 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.125983 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126074 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126118 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126186 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126325 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126243 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126329 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126368 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.126552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.126432 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\") pod \"498528ef-a903-48ac-8a61-4384457579ec\" (UID: \"498528ef-a903-48ac-8a61-4384457579ec\") " Oct 30 05:37:42.128160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.127067 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:42.128160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.127173 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:42.128160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.127511 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:42.128160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.127817 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:42.128160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.127977 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:42.129138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.129106 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:42.129382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.129358 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:37:42.129541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.129516 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "kube-api-access-sfg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:42.129839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.129816 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token" pod "498528ef-a903-48ac-8a61-4384457579ec" (UID: "498528ef-a903-48ac-8a61-4384457579ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228592 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228626 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228640 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228653 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228685 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498528ef-a903-48ac-8a61-4384457579ec-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228712 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfg4q\" (UniqueName: \"kubernetes.io/projected/498528ef-a903-48ac-8a61-4384457579ec-kube-api-access-sfg4q\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228728 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498528ef-a903-48ac-8a61-4384457579ec-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228741 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.228771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.228750 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498528ef-a903-48ac-8a61-4384457579ec-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:42.396042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.396008 5058 generic.go:356] "Generic (PLEG): container finished" podID="498528ef-a903-48ac-8a61-4384457579ec" containerID="930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb" exitCode=0 Oct 30 05:37:42.396545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.396056 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" Oct 30 05:37:42.396545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.396098 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" event={"ID":"498528ef-a903-48ac-8a61-4384457579ec","Type":"ContainerDied","Data":"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb"} Oct 30 05:37:42.396545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.396161 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m" event={"ID":"498528ef-a903-48ac-8a61-4384457579ec","Type":"ContainerDied","Data":"d328ca4d4e925fbc4ff2fbffaf6cdb8d5b57a8adbdf9777947c1911c5590f1f8"} Oct 30 05:37:42.396545 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.396196 5058 scope.go:122] "RemoveContainer" containerID="930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb" Oct 30 05:37:42.412528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.412508 5058 scope.go:122] "RemoveContainer" containerID="930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb" Oct 30 05:37:42.412820 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:42.412803 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb\": container with ID starting with 930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb not found: ID does not exist" containerID="930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb" Oct 30 05:37:42.412894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.412830 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb"} err="failed to get container status \"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb\": rpc error: code = NotFound desc = could not find container \"930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb\": container with ID starting with 930183cc9db943231a514b12266ddded24e605b167735229bd21df62571b5adb not found: ID does not exist" Oct 30 05:37:42.458333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.458300 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:37:42.471620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:42.471589 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m"] Oct 30 05:37:43.162459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:43.162426 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498528ef-a903-48ac-8a61-4384457579ec" path="/var/lib/kubelet/pods/498528ef-a903-48ac-8a61-4384457579ec/volumes" Oct 30 05:37:44.506404 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:44.506365 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:37:44.506820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:44.506365 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:37:46.431008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.430770 5058 generic.go:356] "Generic (PLEG): container finished" podID="6ba47046-1aca-4cff-afe0-56396502592a" containerID="32d7486602eb13f84fe0055143cb9c1e6eb7312d2586d0285401436560e91ac3" exitCode=0 Oct 30 05:37:46.431008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.430883 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" event={"ID":"6ba47046-1aca-4cff-afe0-56396502592a","Type":"ContainerDied","Data":"32d7486602eb13f84fe0055143cb9c1e6eb7312d2586d0285401436560e91ac3"} Oct 30 05:37:46.433461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.433356 5058 generic.go:356] "Generic (PLEG): container finished" podID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerID="45aff2d006b7503ef4765a8b4af5fdacd42070e0db9d0a2424b161cae7d96c05" exitCode=0 Oct 30 05:37:46.433461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.433408 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" event={"ID":"6fc5c8fa-163f-4833-b402-8ea2a285b71a","Type":"ContainerDied","Data":"45aff2d006b7503ef4765a8b4af5fdacd42070e0db9d0a2424b161cae7d96c05"} Oct 30 05:37:46.484483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.484447 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:37:46.507729 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.507701 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\") pod \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " Oct 30 05:37:46.507939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.507816 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\" (UniqueName: \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\") pod \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\" (UID: \"6fc5c8fa-163f-4833-b402-8ea2a285b71a\") " Oct 30 05:37:46.508166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.508142 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume" pod "6fc5c8fa-163f-4833-b402-8ea2a285b71a" (UID: "6fc5c8fa-163f-4833-b402-8ea2a285b71a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:46.509919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.509895 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4" pod "6fc5c8fa-163f-4833-b402-8ea2a285b71a" (UID: "6fc5c8fa-163f-4833-b402-8ea2a285b71a"). InnerVolumeSpecName "kube-api-access-9dxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:46.541514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.541460 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:37:46.608703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.608668 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\" (UniqueName: \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\") pod \"6ba47046-1aca-4cff-afe0-56396502592a\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " Oct 30 05:37:46.608949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.608929 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\") pod \"6ba47046-1aca-4cff-afe0-56396502592a\" (UID: \"6ba47046-1aca-4cff-afe0-56396502592a\") " Oct 30 05:37:46.609416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.609382 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume" pod "6ba47046-1aca-4cff-afe0-56396502592a" (UID: "6ba47046-1aca-4cff-afe0-56396502592a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:46.609416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.609407 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxd4\" (UniqueName: \"kubernetes.io/projected/6fc5c8fa-163f-4833-b402-8ea2a285b71a-kube-api-access-9dxd4\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:46.609579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.609434 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc5c8fa-163f-4833-b402-8ea2a285b71a-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:46.611287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.611259 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l" pod "6ba47046-1aca-4cff-afe0-56396502592a" (UID: "6ba47046-1aca-4cff-afe0-56396502592a"). InnerVolumeSpecName "kube-api-access-rg85l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:46.711166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.711073 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba47046-1aca-4cff-afe0-56396502592a-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:46.711166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.711118 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg85l\" (UniqueName: \"kubernetes.io/projected/6ba47046-1aca-4cff-afe0-56396502592a-kube-api-access-rg85l\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:46.984141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.984064 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:37:46.984384 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:46.984337 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" podUID="e4857400-6961-4a25-9057-a70946e58b9b" containerName="maas-api" containerID="cri-o://c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632" gracePeriod=30 Oct 30 05:37:47.390688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.390657 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:37:47.441725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.441692 5058 generic.go:356] "Generic (PLEG): container finished" podID="e4857400-6961-4a25-9057-a70946e58b9b" containerID="c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632" exitCode=0 Oct 30 05:37:47.442194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.441739 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" Oct 30 05:37:47.442194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.441775 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" event={"ID":"e4857400-6961-4a25-9057-a70946e58b9b","Type":"ContainerDied","Data":"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632"} Oct 30 05:37:47.442194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.441828 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm" event={"ID":"e4857400-6961-4a25-9057-a70946e58b9b","Type":"ContainerDied","Data":"ac1ad89f0d357464c9d25a209e4ca088ac1f80dc294519332929cb7c940f2229"} Oct 30 05:37:47.442194 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.441850 5058 scope.go:122] "RemoveContainer" containerID="c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632" Oct 30 05:37:47.443346 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.443331 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" Oct 30 05:37:47.443423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.443365 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl" event={"ID":"6ba47046-1aca-4cff-afe0-56396502592a","Type":"ContainerDied","Data":"5f5836e2eac0304d03a7d44404eefbf1e4ebfe4ced86932678e8ddd765d0a046"} Oct 30 05:37:47.444935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.444915 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" event={"ID":"6fc5c8fa-163f-4833-b402-8ea2a285b71a","Type":"ContainerDied","Data":"f733c914bf3a4334a073fb6e0af91fb4db875173209a9d7945f52f6ab8870aa1"} Oct 30 05:37:47.445041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.444940 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd" Oct 30 05:37:47.455546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.455520 5058 scope.go:122] "RemoveContainer" containerID="c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632" Oct 30 05:37:47.455930 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:47.455903 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632\": container with ID starting with c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632 not found: ID does not exist" containerID="c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632" Oct 30 05:37:47.456002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.455942 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632"} err="failed to get container status \"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632\": rpc error: code = NotFound desc = could not find container \"c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632\": container with ID starting with c6d9bb60472985f46ec2c9878e705e1369732a87b49ebf41c48cfc033ef87632 not found: ID does not exist" Oct 30 05:37:47.456002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.455969 5058 scope.go:122] "RemoveContainer" containerID="32d7486602eb13f84fe0055143cb9c1e6eb7312d2586d0285401436560e91ac3" Oct 30 05:37:47.469367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.469337 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\") pod \"e4857400-6961-4a25-9057-a70946e58b9b\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " Oct 30 05:37:47.470347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.469416 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\") pod \"e4857400-6961-4a25-9057-a70946e58b9b\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " Oct 30 05:37:47.470347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.469499 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\") pod \"e4857400-6961-4a25-9057-a70946e58b9b\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " Oct 30 05:37:47.470347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.469702 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\" (UniqueName: \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\") pod \"e4857400-6961-4a25-9057-a70946e58b9b\" (UID: \"e4857400-6961-4a25-9057-a70946e58b9b\") " Oct 30 05:37:47.470347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.469768 5058 scope.go:122] "RemoveContainer" containerID="45aff2d006b7503ef4765a8b4af5fdacd42070e0db9d0a2424b161cae7d96c05" Oct 30 05:37:47.471051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.470969 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca" pod "e4857400-6961-4a25-9057-a70946e58b9b" (UID: "e4857400-6961-4a25-9057-a70946e58b9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:47.473441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.473416 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs" pod "e4857400-6961-4a25-9057-a70946e58b9b" (UID: "e4857400-6961-4a25-9057-a70946e58b9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:47.476910 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.476884 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls" pod "e4857400-6961-4a25-9057-a70946e58b9b" (UID: "e4857400-6961-4a25-9057-a70946e58b9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:47.478568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.478546 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx" pod "e4857400-6961-4a25-9057-a70946e58b9b" (UID: "e4857400-6961-4a25-9057-a70946e58b9b"). InnerVolumeSpecName "kube-api-access-6jntx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:47.552840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.552750 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:37:47.556328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.556301 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl"] Oct 30 05:37:47.571636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.571602 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jntx\" (UniqueName: \"kubernetes.io/projected/e4857400-6961-4a25-9057-a70946e58b9b-kube-api-access-6jntx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:47.571636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.571634 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4857400-6961-4a25-9057-a70946e58b9b-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:47.571844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.571650 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:47.571844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.571661 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4857400-6961-4a25-9057-a70946e58b9b-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:37:47.597721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.597688 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:37:47.604837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.604803 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd"] Oct 30 05:37:47.802132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.802094 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:37:47.805528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:47.805447 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44bc69-855f99df97-tktlm"] Oct 30 05:37:49.161321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:49.161286 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba47046-1aca-4cff-afe0-56396502592a" path="/var/lib/kubelet/pods/6ba47046-1aca-4cff-afe0-56396502592a/volumes" Oct 30 05:37:49.161783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:49.161766 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" path="/var/lib/kubelet/pods/6fc5c8fa-163f-4833-b402-8ea2a285b71a/volumes" Oct 30 05:37:49.162276 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:49.162241 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4857400-6961-4a25-9057-a70946e58b9b" path="/var/lib/kubelet/pods/e4857400-6961-4a25-9057-a70946e58b9b/volumes" Oct 30 05:37:49.191178 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:37:49.191143 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:37:56.323484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:56.323452 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:37:56.324141 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:56.323685 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" containerID="cri-o://82d4c0e75fca3cab04623c81a56a2e08d068cac43c3900efdcd8957a6bbfd2af" gracePeriod=30 Oct 30 05:37:57.368715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:57.368680 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:37:57.369531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:57.369496 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" containerID="cri-o://5c148558e63780fdb591852e220305ae9c6e44d50bec3e1febe3cbc077e886be" gracePeriod=30 Oct 30 05:37:58.938063 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:58.938022 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:37:58.938533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:37:58.938115 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:01.577605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.577575 5058 generic.go:356] "Generic (PLEG): container finished" podID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerID="82d4c0e75fca3cab04623c81a56a2e08d068cac43c3900efdcd8957a6bbfd2af" exitCode=0 Oct 30 05:38:01.578032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.577626 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerDied","Data":"82d4c0e75fca3cab04623c81a56a2e08d068cac43c3900efdcd8957a6bbfd2af"} Oct 30 05:38:01.578032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.577704 5058 scope.go:122] "RemoveContainer" containerID="e5e341063ceb9880064af3f2334fe534768fdf90d16f13619f3886ac02cbd8f9" Oct 30 05:38:01.616109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.616086 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:38:01.752455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752423 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.752654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752475 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.752654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752502 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.752654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752580 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.752654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752621 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752720 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752794 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752896 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752954 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\") pod \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\" (UID: \"049cc2ae-2247-43f9-8b0c-7ffba3fe7170\") " Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.752985 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.753118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753011 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:01.753529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753163 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.753529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753391 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.753659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753569 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.753659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753585 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.753659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753596 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.753659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753605 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.753919 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.753896 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.755324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.755296 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:01.755426 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.755354 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:01.755499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.755435 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.755727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.755705 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7" pod "049cc2ae-2247-43f9-8b0c-7ffba3fe7170" (UID: "049cc2ae-2247-43f9-8b0c-7ffba3fe7170"). InnerVolumeSpecName "kube-api-access-b4sk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:01.855143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.855101 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4sk7\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-kube-api-access-b4sk7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.855143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.855139 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.855143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.855152 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.855427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.855164 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.855427 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:01.855177 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049cc2ae-2247-43f9-8b0c-7ffba3fe7170-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.589535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.589493 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" event={"ID":"049cc2ae-2247-43f9-8b0c-7ffba3fe7170","Type":"ContainerDied","Data":"dac6879dc4c7e583a1db405ac2d40b09bd99d5d1aaeee39e89c0128c55da0302"} Oct 30 05:38:02.590034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.589553 5058 scope.go:122] "RemoveContainer" containerID="82d4c0e75fca3cab04623c81a56a2e08d068cac43c3900efdcd8957a6bbfd2af" Oct 30 05:38:02.590034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.589646 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c" Oct 30 05:38:02.593706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.593662 5058 generic.go:356] "Generic (PLEG): container finished" podID="25d26a26-b1d8-4390-8883-7965d19497d4" containerID="5c148558e63780fdb591852e220305ae9c6e44d50bec3e1febe3cbc077e886be" exitCode=0 Oct 30 05:38:02.593863 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.593690 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerDied","Data":"5c148558e63780fdb591852e220305ae9c6e44d50bec3e1febe3cbc077e886be"} Oct 30 05:38:02.602884 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.602727 5058 scope.go:122] "RemoveContainer" containerID="4adb03779991e646532b9b1da0bfef5007433f444adc4ed5a573949b7281da85" Oct 30 05:38:02.662124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.662092 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:38:02.768147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768097 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768154 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768234 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768366 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768485 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768529 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:02.768644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768543 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768625 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:02.768813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768666 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.768813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768736 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:02.768979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.768954 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:02.769135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.769050 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.769376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.769353 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\") pod \"25d26a26-b1d8-4390-8883-7965d19497d4\" (UID: \"25d26a26-b1d8-4390-8883-7965d19497d4\") " Oct 30 05:38:02.769801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.769773 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:02.770124 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.770104 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.770202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.770131 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.770202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.770144 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d26a26-b1d8-4390-8883-7965d19497d4-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.770202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.770156 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.770202 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.770169 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.771051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.771010 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:02.771421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.771395 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:02.771699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.771675 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "kube-api-access-9zwx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:02.771780 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.771678 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy" pod "25d26a26-b1d8-4390-8883-7965d19497d4" (UID: "25d26a26-b1d8-4390-8883-7965d19497d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:02.807334 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.807292 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:38:02.837875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.837834 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c"] Oct 30 05:38:02.871351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.871320 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.871351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.871348 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zwx6\" (UniqueName: \"kubernetes.io/projected/25d26a26-b1d8-4390-8883-7965d19497d4-kube-api-access-9zwx6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.871351 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.871359 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d26a26-b1d8-4390-8883-7965d19497d4-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:02.871599 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:02.871370 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d26a26-b1d8-4390-8883-7965d19497d4-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:03.163669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.163634 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" path="/var/lib/kubelet/pods/049cc2ae-2247-43f9-8b0c-7ffba3fe7170/volumes" Oct 30 05:38:03.602444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.602367 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" Oct 30 05:38:03.602444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.602407 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg" event={"ID":"25d26a26-b1d8-4390-8883-7965d19497d4","Type":"ContainerDied","Data":"5d1c573d13791040ea7fb98ca74c9d1654b20b20eaef047d3f9bd7f8e1e2ba52"} Oct 30 05:38:03.602987 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.602467 5058 scope.go:122] "RemoveContainer" containerID="5c148558e63780fdb591852e220305ae9c6e44d50bec3e1febe3cbc077e886be" Oct 30 05:38:03.681081 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.680786 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:38:03.691772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:03.691731 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg"] Oct 30 05:38:05.162131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:05.162090 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" path="/var/lib/kubelet/pods/25d26a26-b1d8-4390-8883-7965d19497d4/volumes" Oct 30 05:38:07.363412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:07.363378 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:38:09.437050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437013 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:38:09.437693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437677 5058 cpu_manager.go:414] "containerMap: removing container" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437694 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437705 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437710 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437731 5058 cpu_manager.go:414] "containerMap: removing container" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437736 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.437742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437744 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e4857400-6961-4a25-9057-a70946e58b9b" containerName="maas-api" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437749 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4857400-6961-4a25-9057-a70946e58b9b" containerName="maas-api" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437757 5058 cpu_manager.go:414] "containerMap: removing container" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437762 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437771 5058 cpu_manager.go:414] "containerMap: removing container" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437775 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437792 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437796 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437807 5058 cpu_manager.go:414] "containerMap: removing container" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437813 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437823 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437828 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437837 5058 cpu_manager.go:414] "containerMap: removing container" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437842 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437923 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437930 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437938 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437947 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="498528ef-a903-48ac-8a61-4384457579ec" containerName="istio-proxy" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437952 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba47046-1aca-4cff-afe0-56396502592a" containerName="payload-pre-processing" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437961 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4857400-6961-4a25-9057-a70946e58b9b" containerName="maas-api" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437966 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc5c8fa-163f-4833-b402-8ea2a285b71a" containerName="payload-processing" Oct 30 05:38:09.437994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.437972 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="049cc2ae-2247-43f9-8b0c-7ffba3fe7170" containerName="istio-proxy" Oct 30 05:38:09.442938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.442917 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.466044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.466012 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.466252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.466186 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdrb\" (UniqueName: \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.473833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.473804 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a62fee\"" Oct 30 05:38:09.474006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.473899 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a62fee-dockercfg-nwtjj\"" Oct 30 05:38:09.479313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.479264 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:38:09.574280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.573259 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.574280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.573415 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdrb\" (UniqueName: \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.575455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.575430 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.578889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.578864 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:38:09.586965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.586927 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43bbdb4-3917-460e-8c18-d81926232c43" containerName="istio-proxy" Oct 30 05:38:09.587172 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.587155 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d26a26-b1d8-4390-8883-7965d19497d4" containerName="istio-proxy" Oct 30 05:38:09.594751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.594725 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.629211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.629053 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:38:09.648367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.648334 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdrb\" (UniqueName: \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.675706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.675636 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99n2\" (UniqueName: \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.676115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.676087 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.757705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.757617 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:09.777458 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.777426 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99n2\" (UniqueName: \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.777616 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.777592 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.778395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.778371 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.799921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.799881 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99n2\" (UniqueName: \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\") pod \"payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:09.972731 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:09.972691 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:10.072490 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:38:10.072450 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496f5f36_88fe_42e6_b36c_d564ccc16717.slice/crio-2feb33a0e5c951ab88600b54767e5398f16762d038e2c68deaa016731b1f204d WatchSource:0}: Error finding container 2feb33a0e5c951ab88600b54767e5398f16762d038e2c68deaa016731b1f204d: Status 404 returned error can't find the container with id 2feb33a0e5c951ab88600b54767e5398f16762d038e2c68deaa016731b1f204d Oct 30 05:38:10.072718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.072693 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:38:10.157763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.157726 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:38:10.158507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.158171 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" podUID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" containerName="payload-pre-processing" containerID="cri-o://8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4" gracePeriod=30 Oct 30 05:38:10.280671 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:38:10.280643 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abb0e66_ebc5_4a6a_97e6_852adf1887c0.slice/crio-786f74e8ab0d679ffe22616579ca4d0f9c65469b69f739571a2cc691ee255828 WatchSource:0}: Error finding container 786f74e8ab0d679ffe22616579ca4d0f9c65469b69f739571a2cc691ee255828: Status 404 returned error can't find the container with id 786f74e8ab0d679ffe22616579ca4d0f9c65469b69f739571a2cc691ee255828 Oct 30 05:38:10.282204 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.282181 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:38:10.584368 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.584275 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:38:10.667492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.667458 5058 generic.go:356] "Generic (PLEG): container finished" podID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" containerID="8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4" exitCode=0 Oct 30 05:38:10.667707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.667554 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" event={"ID":"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a","Type":"ContainerDied","Data":"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4"} Oct 30 05:38:10.667707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.667591 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" event={"ID":"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a","Type":"ContainerDied","Data":"3e80451d92a220f8184c1e9685b646390d174025d20cbcdca8889311a68d32ed"} Oct 30 05:38:10.667707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.667614 5058 scope.go:122] "RemoveContainer" containerID="8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4" Oct 30 05:38:10.667904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.667721 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8" Oct 30 05:38:10.670205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.670169 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" event={"ID":"8abb0e66-ebc5-4a6a-97e6-852adf1887c0","Type":"ContainerStarted","Data":"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9"} Oct 30 05:38:10.670333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.670266 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:10.670333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.670290 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" event={"ID":"8abb0e66-ebc5-4a6a-97e6-852adf1887c0","Type":"ContainerStarted","Data":"786f74e8ab0d679ffe22616579ca4d0f9c65469b69f739571a2cc691ee255828"} Oct 30 05:38:10.671879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.671854 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" event={"ID":"496f5f36-88fe-42e6-b36c-d564ccc16717","Type":"ContainerStarted","Data":"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e"} Oct 30 05:38:10.671984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.671900 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" event={"ID":"496f5f36-88fe-42e6-b36c-d564ccc16717","Type":"ContainerStarted","Data":"2feb33a0e5c951ab88600b54767e5398f16762d038e2c68deaa016731b1f204d"} Oct 30 05:38:10.671984 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.671928 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:10.683514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.683474 5058 scope.go:122] "RemoveContainer" containerID="8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4" Oct 30 05:38:10.683910 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:10.683873 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4\": container with ID starting with 8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4 not found: ID does not exist" containerID="8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4" Oct 30 05:38:10.684037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.683913 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4"} err="failed to get container status \"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4\": rpc error: code = NotFound desc = could not find container \"8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4\": container with ID starting with 8643c5c14d840398a8f56cd007010303d203c4a32ba92b5f5a56cd39eb87e4e4 not found: ID does not exist" Oct 30 05:38:10.701562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.701531 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\" (UniqueName: \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\") pod \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " Oct 30 05:38:10.701832 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.701809 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\") pod \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\" (UID: \"7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a\") " Oct 30 05:38:10.702253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.702213 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume" pod "7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" (UID: "7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:10.702376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.702349 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.703813 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.703783 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg" pod "7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" (UID: "7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a"). InnerVolumeSpecName "kube-api-access-jp9bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:10.739613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.738867 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" podStartSLOduration=1.738840707 podStartE2EDuration="1.738840707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:10.732004126 +0000 UTC m=+3263.701499108" watchObservedRunningTime="2026-10-30 05:38:10.738840707 +0000 UTC m=+3263.708335697" Oct 30 05:38:10.785712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.785653 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" podStartSLOduration=1.785633744 podStartE2EDuration="1.785633744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:10.78423395 +0000 UTC m=+3263.753728960" watchObservedRunningTime="2026-10-30 05:38:10.785633744 +0000 UTC m=+3263.755128734" Oct 30 05:38:10.803888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:10.803852 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9bg\" (UniqueName: \"kubernetes.io/projected/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a-kube-api-access-jp9bg\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:11.021717 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:11.021674 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:38:11.027741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:11.027702 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8"] Oct 30 05:38:11.162727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:11.162686 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" path="/var/lib/kubelet/pods/7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a/volumes" Oct 30 05:38:12.364144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:12.364106 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" containerID="cri-o://5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b" gracePeriod=25 Oct 30 05:38:14.081941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:14.081905 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:38:19.192201 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:19.192135 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:38:21.683051 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:21.683007 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:38:21.683504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:21.683080 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:38:24.081392 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:24.081358 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:38:32.663872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.663840 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:38:32.807321 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.807280 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\" (UniqueName: \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\") pod \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " Oct 30 05:38:32.807512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.807422 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\") pod \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\" (UID: \"f60928a7-7924-46f2-a3a9-bc75a4fcc018\") " Oct 30 05:38:32.807808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.807784 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume" pod "f60928a7-7924-46f2-a3a9-bc75a4fcc018" (UID: "f60928a7-7924-46f2-a3a9-bc75a4fcc018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:32.809390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.809364 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn" pod "f60928a7-7924-46f2-a3a9-bc75a4fcc018" (UID: "f60928a7-7924-46f2-a3a9-bc75a4fcc018"). InnerVolumeSpecName "kube-api-access-5nfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:32.859052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.859015 5058 generic.go:356] "Generic (PLEG): container finished" podID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerID="5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b" exitCode=0 Oct 30 05:38:32.859252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.859061 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" Oct 30 05:38:32.859252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.859104 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" event={"ID":"f60928a7-7924-46f2-a3a9-bc75a4fcc018","Type":"ContainerDied","Data":"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b"} Oct 30 05:38:32.859252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.859193 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk" event={"ID":"f60928a7-7924-46f2-a3a9-bc75a4fcc018","Type":"ContainerDied","Data":"c725becf53cc49e8c6a39b740827aef8bbcd7d9f112586fb21f36a6fa31b1afb"} Oct 30 05:38:32.859252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.859237 5058 scope.go:122] "RemoveContainer" containerID="5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b" Oct 30 05:38:32.869531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.869513 5058 scope.go:122] "RemoveContainer" containerID="5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b" Oct 30 05:38:32.869807 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:32.869784 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b\": container with ID starting with 5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b not found: ID does not exist" containerID="5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b" Oct 30 05:38:32.869888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.869820 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b"} err="failed to get container status \"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b\": rpc error: code = NotFound desc = could not find container \"5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b\": container with ID starting with 5387fe76d5ca02bd5fce52817ab37be409f5bb5ae60bbd5e9da9872592d2764b not found: ID does not exist" Oct 30 05:38:32.908857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.908829 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f60928a7-7924-46f2-a3a9-bc75a4fcc018-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:32.908857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.908858 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfhn\" (UniqueName: \"kubernetes.io/projected/f60928a7-7924-46f2-a3a9-bc75a4fcc018-kube-api-access-5nfhn\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:32.925601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.925573 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:38:32.931965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:32.931939 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk"] Oct 30 05:38:33.161715 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:33.161638 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" path="/var/lib/kubelet/pods/f60928a7-7924-46f2-a3a9-bc75a4fcc018/volumes" Oct 30 05:38:37.585654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:37.585617 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:38:37.586137 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:37.585820 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" containerID="cri-o://f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff" gracePeriod=30 Oct 30 05:38:37.599565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:37.599532 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:38:37.599864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:37.599833 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" containerID="cri-o://d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e" gracePeriod=30 Oct 30 05:38:41.243413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.243375 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:38:41.243841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.243563 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" podUID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" containerName="payload-processing" containerID="cri-o://9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede" gracePeriod=30 Oct 30 05:38:41.285687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.285649 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:38:41.285956 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.285926 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" podUID="36ad3d95-f894-40d1-a077-488e4250b79c" containerName="payload-pre-processing" containerID="cri-o://046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9" gracePeriod=30 Oct 30 05:38:41.606333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.605469 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:38:41.645279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.642755 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:38:41.645279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.643024 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" podUID="adb747f5-5af4-4af4-b359-92de31f91784" containerName="maas-api" containerID="cri-o://9bb4b7e171faec4ee0ac3d0acc67aac45d682445ca0f0d6146203b1b95d80c1c" gracePeriod=30 Oct 30 05:38:41.719367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.718095 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:38:41.748265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.746424 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\" (UniqueName: \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\") pod \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " Oct 30 05:38:41.748265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.746503 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\") pod \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\" (UID: \"3e5308d7-6e00-4944-ab33-00c1a59a27b2\") " Oct 30 05:38:41.748265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.747101 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume" pod "3e5308d7-6e00-4944-ab33-00c1a59a27b2" (UID: "3e5308d7-6e00-4944-ab33-00c1a59a27b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:41.748265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.747773 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5308d7-6e00-4944-ab33-00c1a59a27b2-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.751055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.750505 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2" pod "3e5308d7-6e00-4944-ab33-00c1a59a27b2" (UID: "3e5308d7-6e00-4944-ab33-00c1a59a27b2"). InnerVolumeSpecName "kube-api-access-w4kj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:41.863282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.856568 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\" (UniqueName: \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\") pod \"36ad3d95-f894-40d1-a077-488e4250b79c\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " Oct 30 05:38:41.863282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.856746 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\") pod \"36ad3d95-f894-40d1-a077-488e4250b79c\" (UID: \"36ad3d95-f894-40d1-a077-488e4250b79c\") " Oct 30 05:38:41.863282 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.857268 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4kj2\" (UniqueName: \"kubernetes.io/projected/3e5308d7-6e00-4944-ab33-00c1a59a27b2-kube-api-access-w4kj2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.878959 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.868526 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume" pod "36ad3d95-f894-40d1-a077-488e4250b79c" (UID: "36ad3d95-f894-40d1-a077-488e4250b79c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:41.887236 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.886254 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j" pod "36ad3d95-f894-40d1-a077-488e4250b79c" (UID: "36ad3d95-f894-40d1-a077-488e4250b79c"). InnerVolumeSpecName "kube-api-access-vpn7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:41.922949 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.922916 5058 generic.go:356] "Generic (PLEG): container finished" podID="adb747f5-5af4-4af4-b359-92de31f91784" containerID="9bb4b7e171faec4ee0ac3d0acc67aac45d682445ca0f0d6146203b1b95d80c1c" exitCode=0 Oct 30 05:38:41.923120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.922983 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" event={"ID":"adb747f5-5af4-4af4-b359-92de31f91784","Type":"ContainerDied","Data":"9bb4b7e171faec4ee0ac3d0acc67aac45d682445ca0f0d6146203b1b95d80c1c"} Oct 30 05:38:41.924514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.924489 5058 generic.go:356] "Generic (PLEG): container finished" podID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" containerID="9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede" exitCode=0 Oct 30 05:38:41.924591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.924539 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" Oct 30 05:38:41.924591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.924580 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" event={"ID":"3e5308d7-6e00-4944-ab33-00c1a59a27b2","Type":"ContainerDied","Data":"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede"} Oct 30 05:38:41.924662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.924637 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l" event={"ID":"3e5308d7-6e00-4944-ab33-00c1a59a27b2","Type":"ContainerDied","Data":"4a5e02236da3abfd3704339084e1daa327a7ffaff1f3346e670c233ae8609ff3"} Oct 30 05:38:41.924699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.924667 5058 scope.go:122] "RemoveContainer" containerID="9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede" Oct 30 05:38:41.926447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.926418 5058 generic.go:356] "Generic (PLEG): container finished" podID="36ad3d95-f894-40d1-a077-488e4250b79c" containerID="046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9" exitCode=0 Oct 30 05:38:41.926620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.926469 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" Oct 30 05:38:41.926620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.926465 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" event={"ID":"36ad3d95-f894-40d1-a077-488e4250b79c","Type":"ContainerDied","Data":"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9"} Oct 30 05:38:41.926620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.926531 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr" event={"ID":"36ad3d95-f894-40d1-a077-488e4250b79c","Type":"ContainerDied","Data":"eb714a341db7020fca04187a0a75d756efbec204cd4c796930bfcaa632474913"} Oct 30 05:38:41.953457 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.953423 5058 scope.go:122] "RemoveContainer" containerID="9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede" Oct 30 05:38:41.961886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.961851 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:38:41.962119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.962089 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" podUID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" containerName="payload-pre-processing" containerID="cri-o://818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb" gracePeriod=30 Oct 30 05:38:41.964746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.964708 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ad3d95-f894-40d1-a077-488e4250b79c-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.964746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.964739 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpn7j\" (UniqueName: \"kubernetes.io/projected/36ad3d95-f894-40d1-a077-488e4250b79c-kube-api-access-vpn7j\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.966068 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:41.965847 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede\": container with ID starting with 9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede not found: ID does not exist" containerID="9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede" Oct 30 05:38:41.966068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.965903 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede"} err="failed to get container status \"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede\": rpc error: code = NotFound desc = could not find container \"9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede\": container with ID starting with 9937aaff010aae015edc5f34f565893c398abddfb61c3b6d34064d2911c4bede not found: ID does not exist" Oct 30 05:38:41.966068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:41.965929 5058 scope.go:122] "RemoveContainer" containerID="046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9" Oct 30 05:38:42.050186 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.050164 5058 scope.go:122] "RemoveContainer" containerID="046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9" Oct 30 05:38:42.050602 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:42.050580 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9\": container with ID starting with 046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9 not found: ID does not exist" containerID="046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9" Oct 30 05:38:42.050681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.050613 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9"} err="failed to get container status \"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9\": rpc error: code = NotFound desc = could not find container \"046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9\": container with ID starting with 046867e3381a04c9b0e22e8a730f1dadba69855f9ac732bb80837b5eddb911c9 not found: ID does not exist" Oct 30 05:38:42.084091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.083838 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:38:42.086542 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.086515 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:38:42.090841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.090815 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr"] Oct 30 05:38:42.136923 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.136889 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:38:42.157324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.157292 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l"] Oct 30 05:38:42.275636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.269636 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\") pod \"adb747f5-5af4-4af4-b359-92de31f91784\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " Oct 30 05:38:42.275636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.269702 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") pod \"adb747f5-5af4-4af4-b359-92de31f91784\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " Oct 30 05:38:42.275636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.269747 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\") pod \"adb747f5-5af4-4af4-b359-92de31f91784\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " Oct 30 05:38:42.275636 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.270001 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\" (UniqueName: \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\") pod \"adb747f5-5af4-4af4-b359-92de31f91784\" (UID: \"adb747f5-5af4-4af4-b359-92de31f91784\") " Oct 30 05:38:42.322568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.306386 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca" pod "adb747f5-5af4-4af4-b359-92de31f91784" (UID: "adb747f5-5af4-4af4-b359-92de31f91784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:42.322568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.309107 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs" pod "adb747f5-5af4-4af4-b359-92de31f91784" (UID: "adb747f5-5af4-4af4-b359-92de31f91784"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:42.322568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.315392 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls" pod "adb747f5-5af4-4af4-b359-92de31f91784" (UID: "adb747f5-5af4-4af4-b359-92de31f91784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:42.322568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.315574 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp" pod "adb747f5-5af4-4af4-b359-92de31f91784" (UID: "adb747f5-5af4-4af4-b359-92de31f91784"). InnerVolumeSpecName "kube-api-access-k5pxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:42.373013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.372259 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.373013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.372289 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5pxp\" (UniqueName: \"kubernetes.io/projected/adb747f5-5af4-4af4-b359-92de31f91784-kube-api-access-k5pxp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.373013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.372301 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb747f5-5af4-4af4-b359-92de31f91784-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.373013 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.372310 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb747f5-5af4-4af4-b359-92de31f91784-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.383156 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.383127 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:38:42.574607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.574518 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\" (UniqueName: \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\") pod \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " Oct 30 05:38:42.574607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.574586 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\") pod \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\" (UID: \"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d\") " Oct 30 05:38:42.575032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.575008 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume" pod "9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" (UID: "9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:42.576751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.576727 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5" pod "9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" (UID: "9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d"). InnerVolumeSpecName "kube-api-access-twsl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:42.676238 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.676191 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsl5\" (UniqueName: \"kubernetes.io/projected/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-kube-api-access-twsl5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.676411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.676258 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.851025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.850996 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:38:42.879066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879033 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879113 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879156 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879245 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879319 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879292 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879326 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879362 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879444 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879475 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\") pod \"ce31c438-647b-4639-b235-f2073e8e0467\" (UID: \"ce31c438-647b-4639-b235-f2073e8e0467\") " Oct 30 05:38:42.879843 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879702 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:42.879946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.879922 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:42.880062 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.880045 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.880129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.880068 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce31c438-647b-4639-b235-f2073e8e0467-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.880294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.880274 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:42.880814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.880778 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:42.881076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.881054 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:42.882705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.882669 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:42.882826 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.882721 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "kube-api-access-xxnlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:42.883015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.882993 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:42.883127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.883108 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo" pod "ce31c438-647b-4639-b235-f2073e8e0467" (UID: "ce31c438-647b-4639-b235-f2073e8e0467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:42.936547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.936508 5058 generic.go:356] "Generic (PLEG): container finished" podID="ce31c438-647b-4639-b235-f2073e8e0467" containerID="f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff" exitCode=0 Oct 30 05:38:42.936764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.936571 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" Oct 30 05:38:42.936764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.936630 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" event={"ID":"ce31c438-647b-4639-b235-f2073e8e0467","Type":"ContainerDied","Data":"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff"} Oct 30 05:38:42.936764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.936681 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f" event={"ID":"ce31c438-647b-4639-b235-f2073e8e0467","Type":"ContainerDied","Data":"178aef0bcb63856719f64cb339da5a3dd6f62d30090e04409a3c8dd453a0b9c4"} Oct 30 05:38:42.936764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.936707 5058 scope.go:122] "RemoveContainer" containerID="f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff" Oct 30 05:38:42.938810 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.938787 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:38:42.948290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.947896 5058 generic.go:356] "Generic (PLEG): container finished" podID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" containerID="818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb" exitCode=0 Oct 30 05:38:42.948290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.948035 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" event={"ID":"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d","Type":"ContainerDied","Data":"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb"} Oct 30 05:38:42.948290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.948086 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" event={"ID":"9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d","Type":"ContainerDied","Data":"264d6e8e76b53f94f368184bbc5fe955af6f4494f7f52fc008ef07ab50212962"} Oct 30 05:38:42.948290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.948159 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s" Oct 30 05:38:42.960455 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.960411 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" event={"ID":"adb747f5-5af4-4af4-b359-92de31f91784","Type":"ContainerDied","Data":"4d7b663276cbee775b94a3b29ebc2f83904f4425ebb8bff8c1b7d04b081c3d4e"} Oct 30 05:38:42.960639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:42.960553 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk" Oct 30 05:38:43.012552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.012525 5058 scope.go:122] "RemoveContainer" containerID="f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff" Oct 30 05:38:43.013976 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:43.013942 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff\": container with ID starting with f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff not found: ID does not exist" containerID="f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff" Oct 30 05:38:43.014148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.014122 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff"} err="failed to get container status \"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff\": rpc error: code = NotFound desc = could not find container \"f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff\": container with ID starting with f9a85854ff52c0c20ce7fd0d925c538549bd126afa41f6ef7b11d4a5e96faaff not found: ID does not exist" Oct 30 05:38:43.014273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.014260 5058 scope.go:122] "RemoveContainer" containerID="818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb" Oct 30 05:38:43.019515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.019485 5058 generic.go:356] "Generic (PLEG): container finished" podID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerID="d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e" exitCode=0 Oct 30 05:38:43.019628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.019527 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" event={"ID":"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6","Type":"ContainerDied","Data":"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e"} Oct 30 05:38:43.019628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.019614 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" event={"ID":"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6","Type":"ContainerDied","Data":"9b171de2b0c7c79c593bb5f88f976f5553e13350c9e1b6c1e0693e4a952dae19"} Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.019842 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.020866 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.020935 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021072 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021129 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021164 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021209 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021333 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021398 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.021431 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\") pod \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\" (UID: \"945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6\") " Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022047 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022074 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022089 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxnlc\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-kube-api-access-xxnlc\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022103 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce31c438-647b-4639-b235-f2073e8e0467-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022115 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022131 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce31c438-647b-4639-b235-f2073e8e0467-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022142 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce31c438-647b-4639-b235-f2073e8e0467-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.022504 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.023618 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:43.024160 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.023878 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.025272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.024600 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.025272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.025033 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.027666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.027618 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.027812 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.027706 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "kube-api-access-wkwx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.030361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.030186 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:43.030361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.030329 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token" pod "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" (UID: "945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.036208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.036182 5058 scope.go:122] "RemoveContainer" containerID="818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb" Oct 30 05:38:43.036653 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:43.036618 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb\": container with ID starting with 818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb not found: ID does not exist" containerID="818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb" Oct 30 05:38:43.036751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.036662 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb"} err="failed to get container status \"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb\": rpc error: code = NotFound desc = could not find container \"818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb\": container with ID starting with 818cfb505e09a1486a73ea3c2e0b390d3a2589a97487552d699d29d2eed09acb not found: ID does not exist" Oct 30 05:38:43.036751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.036681 5058 scope.go:122] "RemoveContainer" containerID="9bb4b7e171faec4ee0ac3d0acc67aac45d682445ca0f0d6146203b1b95d80c1c" Oct 30 05:38:43.046765 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.046744 5058 scope.go:122] "RemoveContainer" containerID="d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e" Oct 30 05:38:43.056478 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.056444 5058 scope.go:122] "RemoveContainer" containerID="d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e" Oct 30 05:38:43.056803 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:43.056779 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e\": container with ID starting with d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e not found: ID does not exist" containerID="d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e" Oct 30 05:38:43.056888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.056814 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e"} err="failed to get container status \"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e\": rpc error: code = NotFound desc = could not find container \"d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e\": container with ID starting with d1a1ba3814b3af1a6da96a2367dbf68774ab823e66cb15a3ac34787e0155fd3e not found: ID does not exist" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124012 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124042 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124051 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124061 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124071 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124080 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124088 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124090 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124097 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.124505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.124106 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwx8\" (UniqueName: \"kubernetes.io/projected/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6-kube-api-access-wkwx8\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.161352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.161319 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ad3d95-f894-40d1-a077-488e4250b79c" path="/var/lib/kubelet/pods/36ad3d95-f894-40d1-a077-488e4250b79c/volumes" Oct 30 05:38:43.161802 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.161787 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" path="/var/lib/kubelet/pods/3e5308d7-6e00-4944-ab33-00c1a59a27b2/volumes" Oct 30 05:38:43.162339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.162322 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:38:43.168541 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.168514 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s"] Oct 30 05:38:43.234475 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.234442 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:38:43.247049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.247016 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7226a8-559f468c89-tkqsk"] Oct 30 05:38:43.299047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.299012 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:38:43.310350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.310314 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f"] Oct 30 05:38:43.380815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.380718 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:38:43.400741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:43.400707 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz"] Oct 30 05:38:45.162916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:45.162881 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" path="/var/lib/kubelet/pods/945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6/volumes" Oct 30 05:38:45.163358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:45.163342 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" path="/var/lib/kubelet/pods/9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d/volumes" Oct 30 05:38:45.163744 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:45.163731 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb747f5-5af4-4af4-b359-92de31f91784" path="/var/lib/kubelet/pods/adb747f5-5af4-4af4-b359-92de31f91784/volumes" Oct 30 05:38:45.164488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:45.164472 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce31c438-647b-4639-b235-f2073e8e0467" path="/var/lib/kubelet/pods/ce31c438-647b-4639-b235-f2073e8e0467/volumes" Oct 30 05:38:47.885889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:47.885861 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:38:47.890085 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:47.890062 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:38:49.192038 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:49.192000 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:38:50.235091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:50.235057 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:38:50.235628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:50.235300 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" containerID="cri-o://751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919" gracePeriod=30 Oct 30 05:38:50.491468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:50.491383 5058 scope.go:122] "RemoveContainer" containerID="4f45c6a10f7c87b38d4c7a0af487466883f5f86df6ba6f8c3b68fc5759bce2b9" Oct 30 05:38:54.618513 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:54.618469 5058 patch_prober.go:29] interesting pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:54.619039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:54.618544 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:55.512676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.512603 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:38:55.598176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598140 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598228 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598268 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598522 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598493 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598531 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:55.598576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598543 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598696 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598629 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:55.598754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598696 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598754 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598725 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:55.598864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598760 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598803 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.598864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.598850 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\") pod \"61810822-ab44-48db-8060-240e7c848ca0\" (UID: \"61810822-ab44-48db-8060-240e7c848ca0\") " Oct 30 05:38:55.599269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599036 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:55.599383 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599351 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:55.599468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599445 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.599535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599473 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.599535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599488 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.599535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599501 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.599535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.599516 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61810822-ab44-48db-8060-240e7c848ca0-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.601476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.601260 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "kube-api-access-zqrdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:55.601476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.601409 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:55.601476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.601416 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:55.601476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.601417 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token" pod "61810822-ab44-48db-8060-240e7c848ca0" (UID: "61810822-ab44-48db-8060-240e7c848ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:55.700781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.700739 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61810822-ab44-48db-8060-240e7c848ca0-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.700781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.700777 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrdx\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-kube-api-access-zqrdx\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.701267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.700794 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61810822-ab44-48db-8060-240e7c848ca0-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.701267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.700806 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61810822-ab44-48db-8060-240e7c848ca0-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.952761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.952721 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 05:38:55.952981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:55.952953 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" podUID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" containerName="maas-api" containerID="cri-o://5d0b9ec3a9f512f84ec78a149d43d3de70c5f91de5a1f064479e8266a58e786e" gracePeriod=30 Oct 30 05:38:56.121538 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.121504 5058 generic.go:356] "Generic (PLEG): container finished" podID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" containerID="5d0b9ec3a9f512f84ec78a149d43d3de70c5f91de5a1f064479e8266a58e786e" exitCode=0 Oct 30 05:38:56.121738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.121581 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" event={"ID":"a03b79b2-9a5a-47cf-b430-8562ca63ceef","Type":"ContainerDied","Data":"5d0b9ec3a9f512f84ec78a149d43d3de70c5f91de5a1f064479e8266a58e786e"} Oct 30 05:38:56.123608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.123582 5058 generic.go:356] "Generic (PLEG): container finished" podID="61810822-ab44-48db-8060-240e7c848ca0" containerID="751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919" exitCode=0 Oct 30 05:38:56.123710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.123646 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" Oct 30 05:38:56.123766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.123651 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerDied","Data":"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919"} Oct 30 05:38:56.123801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.123770 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf" event={"ID":"61810822-ab44-48db-8060-240e7c848ca0","Type":"ContainerDied","Data":"4ec2a6d6807c03e2c6588cb62512d90ddf969a6641f32e7a334d156059ded0dc"} Oct 30 05:38:56.123801 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.123792 5058 scope.go:122] "RemoveContainer" containerID="751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919" Oct 30 05:38:56.138579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.138551 5058 scope.go:122] "RemoveContainer" containerID="1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200" Oct 30 05:38:56.155128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.155095 5058 scope.go:122] "RemoveContainer" containerID="751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919" Oct 30 05:38:56.155587 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:56.155535 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919\": container with ID starting with 751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919 not found: ID does not exist" containerID="751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919" Oct 30 05:38:56.155707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.155601 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919"} err="failed to get container status \"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919\": rpc error: code = NotFound desc = could not find container \"751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919\": container with ID starting with 751e0be5d16bbacdd3ce5084a492cd8571178951438904acb4d94df20c85f919 not found: ID does not exist" Oct 30 05:38:56.155707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.155627 5058 scope.go:122] "RemoveContainer" containerID="1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200" Oct 30 05:38:56.155963 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:38:56.155937 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200\": container with ID starting with 1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200 not found: ID does not exist" containerID="1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200" Oct 30 05:38:56.156048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.155973 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200"} err="failed to get container status \"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200\": rpc error: code = NotFound desc = could not find container \"1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200\": container with ID starting with 1bfbea8887e383cb2536687806e2e7acb368a3c178e51728902dc13723f57200 not found: ID does not exist" Oct 30 05:38:56.195920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.195881 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:38:56.199072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.199047 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf"] Oct 30 05:38:56.227148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.220692 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 05:38:56.307359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.307327 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\" (UniqueName: \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\") pod \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " Oct 30 05:38:56.307579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.307419 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") pod \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " Oct 30 05:38:56.307579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.307465 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\") pod \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " Oct 30 05:38:56.307579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.307579 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\") pod \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\" (UID: \"a03b79b2-9a5a-47cf-b430-8562ca63ceef\") " Oct 30 05:38:56.309031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.309003 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca" pod "a03b79b2-9a5a-47cf-b430-8562ca63ceef" (UID: "a03b79b2-9a5a-47cf-b430-8562ca63ceef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:56.310270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.309969 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq" pod "a03b79b2-9a5a-47cf-b430-8562ca63ceef" (UID: "a03b79b2-9a5a-47cf-b430-8562ca63ceef"). InnerVolumeSpecName "kube-api-access-zzmkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:56.310270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.309980 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls" pod "a03b79b2-9a5a-47cf-b430-8562ca63ceef" (UID: "a03b79b2-9a5a-47cf-b430-8562ca63ceef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:56.311311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.311283 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs" pod "a03b79b2-9a5a-47cf-b430-8562ca63ceef" (UID: "a03b79b2-9a5a-47cf-b430-8562ca63ceef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:56.409794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.409759 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:56.409794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.409795 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03b79b2-9a5a-47cf-b430-8562ca63ceef-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:56.410073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.409811 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03b79b2-9a5a-47cf-b430-8562ca63ceef-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:56.410073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:56.409859 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmkq\" (UniqueName: \"kubernetes.io/projected/a03b79b2-9a5a-47cf-b430-8562ca63ceef-kube-api-access-zzmkq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.132710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.132663 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" event={"ID":"a03b79b2-9a5a-47cf-b430-8562ca63ceef","Type":"ContainerDied","Data":"3ca25f488565e7a3bf8b7b3515327e288a44def1772a0e4133fa6e94e1c284ba"} Oct 30 05:38:57.133311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.132785 5058 scope.go:122] "RemoveContainer" containerID="5d0b9ec3a9f512f84ec78a149d43d3de70c5f91de5a1f064479e8266a58e786e" Oct 30 05:38:57.133311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.132680 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874" Oct 30 05:38:57.161778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.161748 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61810822-ab44-48db-8060-240e7c848ca0" path="/var/lib/kubelet/pods/61810822-ab44-48db-8060-240e7c848ca0/volumes" Oct 30 05:38:57.214577 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.214544 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 05:38:57.230857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:57.230812 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b21507-5bff45f56c-tc874"] Oct 30 05:38:59.160649 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:38:59.160617 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" path="/var/lib/kubelet/pods/a03b79b2-9a5a-47cf-b430-8562ca63ceef/volumes" Oct 30 05:39:10.107613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.107569 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:39:10.108357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.107802 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" podUID="5cbda810-6e57-412a-a569-fbfb50859a32" containerName="maas-api" containerID="cri-o://266e85bab3524aa9e7112b52a4f0bf00c617cf84c09a07ea2d0f43a83db416a8" gracePeriod=30 Oct 30 05:39:10.248934 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.248902 5058 generic.go:356] "Generic (PLEG): container finished" podID="5cbda810-6e57-412a-a569-fbfb50859a32" containerID="266e85bab3524aa9e7112b52a4f0bf00c617cf84c09a07ea2d0f43a83db416a8" exitCode=0 Oct 30 05:39:10.249106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.248964 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" event={"ID":"5cbda810-6e57-412a-a569-fbfb50859a32","Type":"ContainerDied","Data":"266e85bab3524aa9e7112b52a4f0bf00c617cf84c09a07ea2d0f43a83db416a8"} Oct 30 05:39:10.307882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.307848 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:39:10.308119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.308089 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" podUID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" containerName="payload-processing" containerID="cri-o://59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9" gracePeriod=30 Oct 30 05:39:10.331328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.331288 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:39:10.331581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.331529 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" podUID="496f5f36-88fe-42e6-b36c-d564ccc16717" containerName="payload-pre-processing" containerID="cri-o://2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e" gracePeriod=30 Oct 30 05:39:10.489335 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.488971 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:39:10.500894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.500851 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\") pod \"5cbda810-6e57-412a-a569-fbfb50859a32\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " Oct 30 05:39:10.501100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.501013 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\") pod \"5cbda810-6e57-412a-a569-fbfb50859a32\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " Oct 30 05:39:10.501100 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.501087 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\" (UniqueName: \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\") pod \"5cbda810-6e57-412a-a569-fbfb50859a32\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " Oct 30 05:39:10.501249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.501192 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\") pod \"5cbda810-6e57-412a-a569-fbfb50859a32\" (UID: \"5cbda810-6e57-412a-a569-fbfb50859a32\") " Oct 30 05:39:10.502500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.502469 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca" pod "5cbda810-6e57-412a-a569-fbfb50859a32" (UID: "5cbda810-6e57-412a-a569-fbfb50859a32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:10.504746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.504716 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs" pod "5cbda810-6e57-412a-a569-fbfb50859a32" (UID: "5cbda810-6e57-412a-a569-fbfb50859a32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:10.505337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.505312 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj" pod "5cbda810-6e57-412a-a569-fbfb50859a32" (UID: "5cbda810-6e57-412a-a569-fbfb50859a32"). InnerVolumeSpecName "kube-api-access-2qndj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:10.506920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.506896 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls" pod "5cbda810-6e57-412a-a569-fbfb50859a32" (UID: "5cbda810-6e57-412a-a569-fbfb50859a32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:10.589920 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.589883 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 05:39:10.590395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.590364 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerName="maas-api" containerID="cri-o://0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d" gracePeriod=30 Oct 30 05:39:10.605073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.604986 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.605073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.605019 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbda810-6e57-412a-a569-fbfb50859a32-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.605073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.605034 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qndj\" (UniqueName: \"kubernetes.io/projected/5cbda810-6e57-412a-a569-fbfb50859a32-kube-api-access-2qndj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.605073 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.605048 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbda810-6e57-412a-a569-fbfb50859a32-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.672552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.672478 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:39:10.705766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.705728 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:39:10.705766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.705750 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\") pod \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " Oct 30 05:39:10.706027 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.705806 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\" (UniqueName: \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\") pod \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\" (UID: \"8abb0e66-ebc5-4a6a-97e6-852adf1887c0\") " Oct 30 05:39:10.706105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.706045 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume" pod "8abb0e66-ebc5-4a6a-97e6-852adf1887c0" (UID: "8abb0e66-ebc5-4a6a-97e6-852adf1887c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:10.706416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.706394 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.707971 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.707947 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2" pod "8abb0e66-ebc5-4a6a-97e6-852adf1887c0" (UID: "8abb0e66-ebc5-4a6a-97e6-852adf1887c0"). InnerVolumeSpecName "kube-api-access-g99n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:10.806943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.806904 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\") pod \"496f5f36-88fe-42e6-b36c-d564ccc16717\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " Oct 30 05:39:10.808075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.806958 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\" (UniqueName: \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\") pod \"496f5f36-88fe-42e6-b36c-d564ccc16717\" (UID: \"496f5f36-88fe-42e6-b36c-d564ccc16717\") " Oct 30 05:39:10.808075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.807413 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume" pod "496f5f36-88fe-42e6-b36c-d564ccc16717" (UID: "496f5f36-88fe-42e6-b36c-d564ccc16717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:10.808075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.807788 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g99n2\" (UniqueName: \"kubernetes.io/projected/8abb0e66-ebc5-4a6a-97e6-852adf1887c0-kube-api-access-g99n2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.808075 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.807813 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496f5f36-88fe-42e6-b36c-d564ccc16717-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.810584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.810205 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb" pod "496f5f36-88fe-42e6-b36c-d564ccc16717" (UID: "496f5f36-88fe-42e6-b36c-d564ccc16717"). InnerVolumeSpecName "kube-api-access-spdrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:10.874324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.874296 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 05:39:10.909233 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909187 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\") pod \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " Oct 30 05:39:10.909432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909261 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\") pod \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " Oct 30 05:39:10.909432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909341 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\") pod \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " Oct 30 05:39:10.909432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909403 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\" (UniqueName: \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\") pod \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\" (UID: \"32c71880-4ac8-4f66-91e3-66f38ca38ea7\") " Oct 30 05:39:10.909725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909697 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca" pod "32c71880-4ac8-4f66-91e3-66f38ca38ea7" (UID: "32c71880-4ac8-4f66-91e3-66f38ca38ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:10.909861 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909841 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c71880-4ac8-4f66-91e3-66f38ca38ea7-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.909935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.909866 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdrb\" (UniqueName: \"kubernetes.io/projected/496f5f36-88fe-42e6-b36c-d564ccc16717-kube-api-access-spdrb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.911454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.911432 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs" pod "32c71880-4ac8-4f66-91e3-66f38ca38ea7" (UID: "32c71880-4ac8-4f66-91e3-66f38ca38ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:10.911560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.911497 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls" pod "32c71880-4ac8-4f66-91e3-66f38ca38ea7" (UID: "32c71880-4ac8-4f66-91e3-66f38ca38ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:10.911560 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:10.911528 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj" pod "32c71880-4ac8-4f66-91e3-66f38ca38ea7" (UID: "32c71880-4ac8-4f66-91e3-66f38ca38ea7"). InnerVolumeSpecName "kube-api-access-mhvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:11.011316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.011212 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:11.011316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.011271 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c71880-4ac8-4f66-91e3-66f38ca38ea7-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:11.011316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.011281 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhvqj\" (UniqueName: \"kubernetes.io/projected/32c71880-4ac8-4f66-91e3-66f38ca38ea7-kube-api-access-mhvqj\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:11.255865 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.255830 5058 generic.go:356] "Generic (PLEG): container finished" podID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" containerID="59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9" exitCode=0 Oct 30 05:39:11.256385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.255882 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" Oct 30 05:39:11.256385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.255937 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" event={"ID":"8abb0e66-ebc5-4a6a-97e6-852adf1887c0","Type":"ContainerDied","Data":"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9"} Oct 30 05:39:11.256385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.255998 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9" event={"ID":"8abb0e66-ebc5-4a6a-97e6-852adf1887c0","Type":"ContainerDied","Data":"786f74e8ab0d679ffe22616579ca4d0f9c65469b69f739571a2cc691ee255828"} Oct 30 05:39:11.256385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.256032 5058 scope.go:122] "RemoveContainer" containerID="59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9" Oct 30 05:39:11.257503 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.257470 5058 generic.go:356] "Generic (PLEG): container finished" podID="496f5f36-88fe-42e6-b36c-d564ccc16717" containerID="2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e" exitCode=0 Oct 30 05:39:11.257621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.257518 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" event={"ID":"496f5f36-88fe-42e6-b36c-d564ccc16717","Type":"ContainerDied","Data":"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e"} Oct 30 05:39:11.257621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.257574 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" event={"ID":"496f5f36-88fe-42e6-b36c-d564ccc16717","Type":"ContainerDied","Data":"2feb33a0e5c951ab88600b54767e5398f16762d038e2c68deaa016731b1f204d"} Oct 30 05:39:11.257621 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.257533 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp" Oct 30 05:39:11.259227 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.259206 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" Oct 30 05:39:11.259361 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.259249 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn" event={"ID":"5cbda810-6e57-412a-a569-fbfb50859a32","Type":"ContainerDied","Data":"4d0e513e7c77350a883741d9de5e790ba72947633825f62d5800c538203662e3"} Oct 30 05:39:11.260869 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.260845 5058 generic.go:356] "Generic (PLEG): container finished" podID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerID="0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d" exitCode=0 Oct 30 05:39:11.260973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.260888 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" event={"ID":"32c71880-4ac8-4f66-91e3-66f38ca38ea7","Type":"ContainerDied","Data":"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d"} Oct 30 05:39:11.260973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.260896 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" Oct 30 05:39:11.260973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.260923 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" event={"ID":"32c71880-4ac8-4f66-91e3-66f38ca38ea7","Type":"ContainerDied","Data":"c08d4b9652e3b3380e7f51dc2ad123bb1e8cbdb1edde67265b397c5f91411ec6"} Oct 30 05:39:11.270288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.270268 5058 scope.go:122] "RemoveContainer" containerID="59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9" Oct 30 05:39:11.270591 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:11.270574 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9\": container with ID starting with 59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9 not found: ID does not exist" containerID="59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9" Oct 30 05:39:11.270643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.270603 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9"} err="failed to get container status \"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9\": rpc error: code = NotFound desc = could not find container \"59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9\": container with ID starting with 59b20c8ea0cc5c4b88dcfe2b580eedf1d9b2b591f7d440cfcd195a610470d4c9 not found: ID does not exist" Oct 30 05:39:11.270643 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.270626 5058 scope.go:122] "RemoveContainer" containerID="2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e" Oct 30 05:39:11.279800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.279780 5058 scope.go:122] "RemoveContainer" containerID="2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e" Oct 30 05:39:11.280107 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:11.280086 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e\": container with ID starting with 2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e not found: ID does not exist" containerID="2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e" Oct 30 05:39:11.280148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.280117 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e"} err="failed to get container status \"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e\": rpc error: code = NotFound desc = could not find container \"2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e\": container with ID starting with 2e876f32f712225f62a5b50b893bf70dbdff24088ede979ecae919d68cb59f7e not found: ID does not exist" Oct 30 05:39:11.280148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.280135 5058 scope.go:122] "RemoveContainer" containerID="266e85bab3524aa9e7112b52a4f0bf00c617cf84c09a07ea2d0f43a83db416a8" Oct 30 05:39:11.289337 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.289317 5058 scope.go:122] "RemoveContainer" containerID="0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d" Oct 30 05:39:11.326790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.326756 5058 scope.go:122] "RemoveContainer" containerID="0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d" Oct 30 05:39:11.331303 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:11.331015 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d\": container with ID starting with 0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d not found: ID does not exist" containerID="0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d" Oct 30 05:39:11.331303 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.331066 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d"} err="failed to get container status \"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d\": rpc error: code = NotFound desc = could not find container \"0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d\": container with ID starting with 0c279c47237810b0d7ad5f9a8cb4240c3e50ef4d912f4ed2b9c554916507d17d not found: ID does not exist" Oct 30 05:39:11.349201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.348436 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:39:11.358901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.358867 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9"] Oct 30 05:39:11.377814 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.377775 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 05:39:11.382106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.382076 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj"] Oct 30 05:39:11.420474 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.420427 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:39:11.425977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.425943 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp"] Oct 30 05:39:11.464249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.464186 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:39:11.467505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:11.467483 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a62fee-7796bd6dc-ct5dn"] Oct 30 05:39:13.160988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:13.160952 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" path="/var/lib/kubelet/pods/32c71880-4ac8-4f66-91e3-66f38ca38ea7/volumes" Oct 30 05:39:13.161423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:13.161358 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496f5f36-88fe-42e6-b36c-d564ccc16717" path="/var/lib/kubelet/pods/496f5f36-88fe-42e6-b36c-d564ccc16717/volumes" Oct 30 05:39:13.161767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:13.161751 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbda810-6e57-412a-a569-fbfb50859a32" path="/var/lib/kubelet/pods/5cbda810-6e57-412a-a569-fbfb50859a32/volumes" Oct 30 05:39:13.162430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:13.162415 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" path="/var/lib/kubelet/pods/8abb0e66-ebc5-4a6a-97e6-852adf1887c0/volumes" Oct 30 05:39:13.793632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:13.793587 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-397179-64d58f6878-r2vvj" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": context deadline exceeded" Oct 30 05:39:18.710243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:18.710177 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:39:18.710867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:18.710489 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" containerID="cri-o://785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39" gracePeriod=30 Oct 30 05:39:19.191004 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:19.190967 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:39:20.262701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:20.262669 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 05:39:20.263178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:20.262858 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" containerID="cri-o://3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2" gracePeriod=30 Oct 30 05:39:20.450889 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:20.450856 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:39:20.451157 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:20.451110 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" containerID="cri-o://e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9" gracePeriod=30 Oct 30 05:39:24.000299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.000271 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:39:24.102031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.101994 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102054 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102116 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102213 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102200 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102295 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102324 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102410 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102449 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102445 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102489 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\") pod \"2f1d7922-e2ef-44cb-9b00-51becde9339f\" (UID: \"2f1d7922-e2ef-44cb-9b00-51becde9339f\") " Oct 30 05:39:24.102668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102501 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:24.102668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102576 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102677 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102721 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.102836 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.103041 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1d7922-e2ef-44cb-9b00-51becde9339f-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.103064 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.103079 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.103099 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.103155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.103118 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.104939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.104916 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "kube-api-access-2dsrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:24.105045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.104943 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:24.105173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.105147 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:24.105411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.105390 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo" pod "2f1d7922-e2ef-44cb-9b00-51becde9339f" (UID: "2f1d7922-e2ef-44cb-9b00-51becde9339f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:24.204197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.204157 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.204197 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.204198 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.204413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.204214 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsrw\" (UniqueName: \"kubernetes.io/projected/2f1d7922-e2ef-44cb-9b00-51becde9339f-kube-api-access-2dsrw\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.204413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.204244 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1d7922-e2ef-44cb-9b00-51becde9339f-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.347043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.346936 5058 generic.go:356] "Generic (PLEG): container finished" podID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerID="785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39" exitCode=0 Oct 30 05:39:24.347043 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.347006 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" Oct 30 05:39:24.347338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.347019 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerDied","Data":"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39"} Oct 30 05:39:24.347338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.347109 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6" event={"ID":"2f1d7922-e2ef-44cb-9b00-51becde9339f","Type":"ContainerDied","Data":"3520f0ca0426e4e571d9ab5e1d107d19c74b2c0036d5e9dfce405f2c1fac817a"} Oct 30 05:39:24.347338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.347153 5058 scope.go:122] "RemoveContainer" containerID="785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39" Oct 30 05:39:24.360430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.360407 5058 scope.go:122] "RemoveContainer" containerID="d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b" Oct 30 05:39:24.369995 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.369969 5058 scope.go:122] "RemoveContainer" containerID="785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39" Oct 30 05:39:24.370361 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:24.370340 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39\": container with ID starting with 785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39 not found: ID does not exist" containerID="785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39" Oct 30 05:39:24.370440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.370370 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39"} err="failed to get container status \"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39\": rpc error: code = NotFound desc = could not find container \"785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39\": container with ID starting with 785e685310be35b01dd9d7c3c108510c619bbc649eeee5330a56703d5a1a7a39 not found: ID does not exist" Oct 30 05:39:24.370440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.370389 5058 scope.go:122] "RemoveContainer" containerID="d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b" Oct 30 05:39:24.370655 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:24.370636 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b\": container with ID starting with d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b not found: ID does not exist" containerID="d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b" Oct 30 05:39:24.370706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.370661 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b"} err="failed to get container status \"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b\": rpc error: code = NotFound desc = could not find container \"d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b\": container with ID starting with d38c9d9daf3eaaf5aaa45746551a1bc6825d9de17c01d59d05130a7d4b80568b not found: ID does not exist" Oct 30 05:39:24.407273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.407240 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:39:24.421274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:24.421206 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6"] Oct 30 05:39:25.162552 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.162517 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" path="/var/lib/kubelet/pods/2f1d7922-e2ef-44cb-9b00-51becde9339f/volumes" Oct 30 05:39:25.582394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.582364 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 05:39:25.703289 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.703260 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:39:25.719695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719629 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719684 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719725 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719787 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719829 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719869 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.719955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.719943 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.720249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720063 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.720249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720131 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\") pod \"7723d7b2-703a-45c2-9a32-a05d048c9d72\" (UID: \"7723d7b2-703a-45c2-9a32-a05d048c9d72\") " Oct 30 05:39:25.720396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720337 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.720470 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720397 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.720530 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720493 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.720588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720570 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.720708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720664 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:25.720708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720701 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.720856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720717 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.720856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720730 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.720856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.720755 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.722593 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.722572 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "kube-api-access-w9ksz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.722790 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.722773 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.722999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.722972 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:25.723095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.723008 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token" pod "7723d7b2-703a-45c2-9a32-a05d048c9d72" (UID: "7723d7b2-703a-45c2-9a32-a05d048c9d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.822047 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822014 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822061 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822131 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822154 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822184 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822323 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822399 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822472 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822512 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\") pod \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\" (UID: \"54b677a9-5b44-4b9f-a853-3d68e2ab3d94\") " Oct 30 05:39:25.822817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822550 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.822817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822564 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.822817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822675 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.822988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822854 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.822988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.822868 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:25.823117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823070 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823096 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823117 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823111 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823124 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823137 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7723d7b2-703a-45c2-9a32-a05d048c9d72-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823150 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823163 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823176 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823191 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9ksz\" (UniqueName: \"kubernetes.io/projected/7723d7b2-703a-45c2-9a32-a05d048c9d72-kube-api-access-w9ksz\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.823311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.823206 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7723d7b2-703a-45c2-9a32-a05d048c9d72-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.824547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.824515 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:25.824661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.824559 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.824661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.824586 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.824661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.824607 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg" pod "54b677a9-5b44-4b9f-a853-3d68e2ab3d94" (UID: "54b677a9-5b44-4b9f-a853-3d68e2ab3d94"). InnerVolumeSpecName "kube-api-access-k9nqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.924687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.924651 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.924687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.924681 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.924687 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.924690 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.924943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:25.924701 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nqg\" (UniqueName: \"kubernetes.io/projected/54b677a9-5b44-4b9f-a853-3d68e2ab3d94-kube-api-access-k9nqg\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:26.363333 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.363290 5058 generic.go:356] "Generic (PLEG): container finished" podID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerID="e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9" exitCode=0 Oct 30 05:39:26.363807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.363332 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" event={"ID":"54b677a9-5b44-4b9f-a853-3d68e2ab3d94","Type":"ContainerDied","Data":"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9"} Oct 30 05:39:26.363807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.363390 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" event={"ID":"54b677a9-5b44-4b9f-a853-3d68e2ab3d94","Type":"ContainerDied","Data":"7052433e2a7c7db50233417efb3e45abaef8909c8e5c840809e41a39cd87a40c"} Oct 30 05:39:26.363807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.363416 5058 scope.go:122] "RemoveContainer" containerID="e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9" Oct 30 05:39:26.363807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.363348 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v" Oct 30 05:39:26.365097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.365049 5058 generic.go:356] "Generic (PLEG): container finished" podID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerID="3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2" exitCode=0 Oct 30 05:39:26.365097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.365082 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" event={"ID":"7723d7b2-703a-45c2-9a32-a05d048c9d72","Type":"ContainerDied","Data":"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2"} Oct 30 05:39:26.365097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.365099 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" Oct 30 05:39:26.365381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.365133 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn" event={"ID":"7723d7b2-703a-45c2-9a32-a05d048c9d72","Type":"ContainerDied","Data":"2ebcae60a67a5a27a7567db77a8ea1d5369742f96993a385b2bc2964f48ecc4c"} Oct 30 05:39:26.374377 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.374352 5058 scope.go:122] "RemoveContainer" containerID="e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9" Oct 30 05:39:26.374688 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:26.374658 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9\": container with ID starting with e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9 not found: ID does not exist" containerID="e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9" Oct 30 05:39:26.374795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.374699 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9"} err="failed to get container status \"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9\": rpc error: code = NotFound desc = could not find container \"e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9\": container with ID starting with e1cec268707bc320b33e592a04237a8a891e17ef8a7331a24d26c64a029dfab9 not found: ID does not exist" Oct 30 05:39:26.374795 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.374725 5058 scope.go:122] "RemoveContainer" containerID="3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2" Oct 30 05:39:26.385121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.384981 5058 scope.go:122] "RemoveContainer" containerID="3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2" Oct 30 05:39:26.385349 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:26.385329 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2\": container with ID starting with 3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2 not found: ID does not exist" containerID="3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2" Oct 30 05:39:26.385400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.385357 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2"} err="failed to get container status \"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2\": rpc error: code = NotFound desc = could not find container \"3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2\": container with ID starting with 3a731ab6ee96f9d17e167127c572b5a05014b1b62ead9303abf97ebacc14f0c2 not found: ID does not exist" Oct 30 05:39:26.421294 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.421257 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 05:39:26.435049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.435014 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn"] Oct 30 05:39:26.466491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.466461 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:39:26.471476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:26.471450 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v"] Oct 30 05:39:27.161581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:27.161548 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" path="/var/lib/kubelet/pods/54b677a9-5b44-4b9f-a853-3d68e2ab3d94/volumes" Oct 30 05:39:27.162015 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:27.162001 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" path="/var/lib/kubelet/pods/7723d7b2-703a-45c2-9a32-a05d048c9d72/volumes" Oct 30 05:39:31.560199 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:31.560165 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:39:34.287709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.287676 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:39:34.288151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.287904 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" podUID="7753b434-1229-4fe8-8848-560519c4028a" containerName="maas-api" containerID="cri-o://6dd369c65e43317f7a57f9d1ab6fd23ed61d2ccfcf00c2a76491bb674a5b0cc0" gracePeriod=30 Oct 30 05:39:34.293012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.292982 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" podUID="7753b434-1229-4fe8-8848-560519c4028a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.189:8443/health\": dial tcp 10.132.0.189:8443: connect: connection refused" Oct 30 05:39:34.419265 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.419199 5058 generic.go:356] "Generic (PLEG): container finished" podID="7753b434-1229-4fe8-8848-560519c4028a" containerID="6dd369c65e43317f7a57f9d1ab6fd23ed61d2ccfcf00c2a76491bb674a5b0cc0" exitCode=0 Oct 30 05:39:34.419408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.419262 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" event={"ID":"7753b434-1229-4fe8-8848-560519c4028a","Type":"ContainerDied","Data":"6dd369c65e43317f7a57f9d1ab6fd23ed61d2ccfcf00c2a76491bb674a5b0cc0"} Oct 30 05:39:34.577605 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.577574 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:39:34.643048 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.643009 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\") pod \"7753b434-1229-4fe8-8848-560519c4028a\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " Oct 30 05:39:34.643280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.643156 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\") pod \"7753b434-1229-4fe8-8848-560519c4028a\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " Oct 30 05:39:34.643280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.643202 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\") pod \"7753b434-1229-4fe8-8848-560519c4028a\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " Oct 30 05:39:34.643400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.643297 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\" (UniqueName: \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\") pod \"7753b434-1229-4fe8-8848-560519c4028a\" (UID: \"7753b434-1229-4fe8-8848-560519c4028a\") " Oct 30 05:39:34.644375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.644332 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca" pod "7753b434-1229-4fe8-8848-560519c4028a" (UID: "7753b434-1229-4fe8-8848-560519c4028a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:34.647752 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.646939 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls" pod "7753b434-1229-4fe8-8848-560519c4028a" (UID: "7753b434-1229-4fe8-8848-560519c4028a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:34.648080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.648058 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6" pod "7753b434-1229-4fe8-8848-560519c4028a" (UID: "7753b434-1229-4fe8-8848-560519c4028a"). InnerVolumeSpecName "kube-api-access-chln6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:34.648775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.648752 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs" pod "7753b434-1229-4fe8-8848-560519c4028a" (UID: "7753b434-1229-4fe8-8848-560519c4028a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:34.755461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.746628 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chln6\" (UniqueName: \"kubernetes.io/projected/7753b434-1229-4fe8-8848-560519c4028a-kube-api-access-chln6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:34.755461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.746719 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:34.755461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.746744 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b434-1229-4fe8-8848-560519c4028a-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:34.755461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:34.746758 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7753b434-1229-4fe8-8848-560519c4028a-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:35.426635 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:35.426604 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" event={"ID":"7753b434-1229-4fe8-8848-560519c4028a","Type":"ContainerDied","Data":"44b643f406562a1eaae927b334d1396e7bc7edacff277f8610a7cc8eca34b451"} Oct 30 05:39:35.427120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:35.426664 5058 scope.go:122] "RemoveContainer" containerID="6dd369c65e43317f7a57f9d1ab6fd23ed61d2ccfcf00c2a76491bb674a5b0cc0" Oct 30 05:39:35.427120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:35.426615 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn" Oct 30 05:39:35.483679 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:35.483646 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:39:35.497125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:35.497095 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0dba31-6849d448d-djngn"] Oct 30 05:39:36.560846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:36.560798 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" containerID="cri-o://98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011" gracePeriod=25 Oct 30 05:39:37.161176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:37.161140 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7753b434-1229-4fe8-8848-560519c4028a" path="/var/lib/kubelet/pods/7753b434-1229-4fe8-8848-560519c4028a/volumes" Oct 30 05:39:38.368261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.368193 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:39:38.369123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.368508 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" podUID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" containerName="maas-api" containerID="cri-o://32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0" gracePeriod=30 Oct 30 05:39:38.637944 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.637360 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:38.675648 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.675622 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:39:38.689597 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.689563 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\") pod \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " Oct 30 05:39:38.689781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.689608 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\" (UniqueName: \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\") pod \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " Oct 30 05:39:38.689781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.689648 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\") pod \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " Oct 30 05:39:38.689899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.689857 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\") pod \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\" (UID: \"2b9f62ec-4c65-45e0-8f4e-725f14779dfb\") " Oct 30 05:39:38.690114 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.690084 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca" pod "2b9f62ec-4c65-45e0-8f4e-725f14779dfb" (UID: "2b9f62ec-4c65-45e0-8f4e-725f14779dfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:38.690328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.690309 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.691955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.691926 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls" pod "2b9f62ec-4c65-45e0-8f4e-725f14779dfb" (UID: "2b9f62ec-4c65-45e0-8f4e-725f14779dfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:38.692326 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.692303 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs" pod "2b9f62ec-4c65-45e0-8f4e-725f14779dfb" (UID: "2b9f62ec-4c65-45e0-8f4e-725f14779dfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:38.692429 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.692356 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q" pod "2b9f62ec-4c65-45e0-8f4e-725f14779dfb" (UID: "2b9f62ec-4c65-45e0-8f4e-725f14779dfb"). InnerVolumeSpecName "kube-api-access-gbt6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:38.791501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.791462 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.791501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.791497 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.791501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:38.791507 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbt6q\" (UniqueName: \"kubernetes.io/projected/2b9f62ec-4c65-45e0-8f4e-725f14779dfb-kube-api-access-gbt6q\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:39.451596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.451506 5058 generic.go:356] "Generic (PLEG): container finished" podID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" containerID="32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0" exitCode=0 Oct 30 05:39:39.451596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.451554 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" Oct 30 05:39:39.452089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.451591 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" event={"ID":"2b9f62ec-4c65-45e0-8f4e-725f14779dfb","Type":"ContainerDied","Data":"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0"} Oct 30 05:39:39.452089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.451648 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7" event={"ID":"2b9f62ec-4c65-45e0-8f4e-725f14779dfb","Type":"ContainerDied","Data":"1ef937edb982193353a3571738cb6da10db16da07dc51eb9570b5c7b9a30fb7a"} Oct 30 05:39:39.452089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.451671 5058 scope.go:122] "RemoveContainer" containerID="32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0" Oct 30 05:39:39.461113 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.461095 5058 scope.go:122] "RemoveContainer" containerID="32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0" Oct 30 05:39:39.461451 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:39.461429 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0\": container with ID starting with 32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0 not found: ID does not exist" containerID="32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0" Oct 30 05:39:39.461546 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.461457 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0"} err="failed to get container status \"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0\": rpc error: code = NotFound desc = could not find container \"32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0\": container with ID starting with 32b2ee6fe19b11c26ccb6298320b0ddb6340c3eb34575c0a22c9dbfb3865a4b0 not found: ID does not exist" Oct 30 05:39:39.510004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.509969 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:39:39.516433 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:39.516404 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35a94e-695d88dbc7-jxpq7"] Oct 30 05:39:41.160788 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:41.160744 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" path="/var/lib/kubelet/pods/2b9f62ec-4c65-45e0-8f4e-725f14779dfb/volumes" Oct 30 05:39:42.776508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:42.776474 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:39:42.777055 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:42.776727 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" containerID="cri-o://baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248" gracePeriod=30 Oct 30 05:39:46.578170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:46.578130 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:39:46.578742 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:46.578465 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" containerID="cri-o://f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3" gracePeriod=30 Oct 30 05:39:46.591842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:46.590910 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:39:46.591842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:46.591155 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" containerID="cri-o://a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d" gracePeriod=30 Oct 30 05:39:48.051674 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.051612 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:39:48.096264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096203 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096314 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096481 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096584 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096634 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:48.096740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096677 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096724 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096752 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096812 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.096862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096841 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.097021 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.096862 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\") pod \"984f6054-dee9-4a26-a54c-e13180ed91b9\" (UID: \"984f6054-dee9-4a26-a54c-e13180ed91b9\") " Oct 30 05:39:48.097338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.097302 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984f6054-dee9-4a26-a54c-e13180ed91b9-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.097338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.097331 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.097543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.097377 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.097543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.097441 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.097543 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.097492 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.099138 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.099114 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:48.099287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.099253 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:48.099620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.099595 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.102393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.102369 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl" pod "984f6054-dee9-4a26-a54c-e13180ed91b9" (UID: "984f6054-dee9-4a26-a54c-e13180ed91b9"). InnerVolumeSpecName "kube-api-access-d96dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:48.198737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198703 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198734 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198747 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198760 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198771 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984f6054-dee9-4a26-a54c-e13180ed91b9-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198783 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96dl\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-kube-api-access-d96dl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.198954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.198794 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984f6054-dee9-4a26-a54c-e13180ed91b9-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.519799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.519768 5058 generic.go:356] "Generic (PLEG): container finished" podID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerID="baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248" exitCode=0 Oct 30 05:39:48.519958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.519816 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" Oct 30 05:39:48.519958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.519850 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" event={"ID":"984f6054-dee9-4a26-a54c-e13180ed91b9","Type":"ContainerDied","Data":"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248"} Oct 30 05:39:48.519958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.519910 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp" event={"ID":"984f6054-dee9-4a26-a54c-e13180ed91b9","Type":"ContainerDied","Data":"8a3f6375c91799f7e8541e5f181b81b63b790cfe5a19ddaabee25d2f363f18d2"} Oct 30 05:39:48.519958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.519937 5058 scope.go:122] "RemoveContainer" containerID="baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248" Oct 30 05:39:48.530428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.530407 5058 scope.go:122] "RemoveContainer" containerID="baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248" Oct 30 05:39:48.530733 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:48.530712 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248\": container with ID starting with baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248 not found: ID does not exist" containerID="baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248" Oct 30 05:39:48.530803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.530742 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248"} err="failed to get container status \"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248\": rpc error: code = NotFound desc = could not find container \"baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248\": container with ID starting with baef63bd4bb6dc3de7bb9e9e3ca520d4db046bc9c4ac387bbe6f96157601a248 not found: ID does not exist" Oct 30 05:39:48.576999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.576962 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:39:48.584940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.584904 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp"] Oct 30 05:39:48.635163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:48.635128 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:49.161022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:49.160988 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" path="/var/lib/kubelet/pods/984f6054-dee9-4a26-a54c-e13180ed91b9/volumes" Oct 30 05:39:49.191087 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:49.191049 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:39:49.265052 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:49.265017 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:39:49.265259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:49.265077 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:39:49.370757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:49.370722 5058 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:39:49.370925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:49.370779 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:39:51.870234 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.870194 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:39:51.891498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.891475 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:39:51.937338 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937303 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937389 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937421 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937466 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937538 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937576 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937638 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:51.937701 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937645 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937782 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937820 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\") pod \"db2d872c-1423-440f-8e42-0bd43c2c4f07\" (UID: \"db2d872c-1423-440f-8e42-0bd43c2c4f07\") " Oct 30 05:39:51.937925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937866 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:51.937925 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.937878 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:51.938229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938092 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:51.938229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938110 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:51.938341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938284 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.938341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938306 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.938341 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938323 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.938509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938345 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d872c-1423-440f-8e42-0bd43c2c4f07-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.938509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.938360 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.940128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.940105 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:51.940277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.940157 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:51.940505 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.940489 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "kube-api-access-zz52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:51.940555 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:51.940512 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token" pod "db2d872c-1423-440f-8e42-0bd43c2c4f07" (UID: "db2d872c-1423-440f-8e42-0bd43c2c4f07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.039661 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039625 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.039867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039704 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.039867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039733 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.039867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039824 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.039867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039852 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.040102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.039945 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.040102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040057 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.040250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040202 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.040330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040276 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:52.040330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040301 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.040440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040344 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.040440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040394 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\") pod \"b885da70-8b9f-47a5-933d-ad47e65bd220\" (UID: \"b885da70-8b9f-47a5-933d-ad47e65bd220\") " Oct 30 05:39:52.040591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040556 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.040724 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040684 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.040875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040856 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz52v\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-kube-api-access-zz52v\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040878 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040895 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040912 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040920 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040927 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040935 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040943 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b885da70-8b9f-47a5-933d-ad47e65bd220-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.040958 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.040951 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d872c-1423-440f-8e42-0bd43c2c4f07-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.042345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.042242 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.042345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.042316 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:52.042557 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.042534 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "kube-api-access-2hhr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.042619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.042602 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy" pod "b885da70-8b9f-47a5-933d-ad47e65bd220" (UID: "b885da70-8b9f-47a5-933d-ad47e65bd220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.142567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.142536 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hhr9\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-kube-api-access-2hhr9\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.142567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.142568 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.142775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.142578 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.142775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.142588 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b885da70-8b9f-47a5-933d-ad47e65bd220-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.550127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.550040 5058 generic.go:356] "Generic (PLEG): container finished" podID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerID="a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d" exitCode=0 Oct 30 05:39:52.550127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.550090 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" Oct 30 05:39:52.550399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.550126 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" event={"ID":"b885da70-8b9f-47a5-933d-ad47e65bd220","Type":"ContainerDied","Data":"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d"} Oct 30 05:39:52.550399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.550179 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm" event={"ID":"b885da70-8b9f-47a5-933d-ad47e65bd220","Type":"ContainerDied","Data":"184d87f07205124796441c292e2dfb9faa3d969b9a80be8c5a8f4b237b045458"} Oct 30 05:39:52.550399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.550211 5058 scope.go:122] "RemoveContainer" containerID="a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d" Oct 30 05:39:52.551641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.551615 5058 generic.go:356] "Generic (PLEG): container finished" podID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerID="f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3" exitCode=0 Oct 30 05:39:52.551755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.551673 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" Oct 30 05:39:52.551755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.551682 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" event={"ID":"db2d872c-1423-440f-8e42-0bd43c2c4f07","Type":"ContainerDied","Data":"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3"} Oct 30 05:39:52.551755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.551720 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768" event={"ID":"db2d872c-1423-440f-8e42-0bd43c2c4f07","Type":"ContainerDied","Data":"2073fb7ad2c06ea0ad5e797d09e10d3e2e84ddee9f2caaaceeeb718be9fbefa0"} Oct 30 05:39:52.561916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.561888 5058 scope.go:122] "RemoveContainer" containerID="a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d" Oct 30 05:39:52.562431 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:52.562407 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d\": container with ID starting with a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d not found: ID does not exist" containerID="a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d" Oct 30 05:39:52.562523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.562438 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d"} err="failed to get container status \"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d\": rpc error: code = NotFound desc = could not find container \"a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d\": container with ID starting with a18ea0489ea3af16113e1b4749a65269f785ed3a36618e8da01fd5cd36a0581d not found: ID does not exist" Oct 30 05:39:52.562523 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.562462 5058 scope.go:122] "RemoveContainer" containerID="f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3" Oct 30 05:39:52.572001 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.571975 5058 scope.go:122] "RemoveContainer" containerID="f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3" Oct 30 05:39:52.572962 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:52.572942 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3\": container with ID starting with f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3 not found: ID does not exist" containerID="f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3" Oct 30 05:39:52.573044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.572970 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3"} err="failed to get container status \"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3\": rpc error: code = NotFound desc = could not find container \"f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3\": container with ID starting with f518949a56285055ee1cac608e4bf9e4ecea2c1f0eb78f6177588e27e24136a3 not found: ID does not exist" Oct 30 05:39:52.608127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.608098 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:39:52.621720 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.621691 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768"] Oct 30 05:39:52.653539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.653489 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:39:52.664148 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:52.664112 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm"] Oct 30 05:39:53.160284 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:53.160252 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" path="/var/lib/kubelet/pods/b885da70-8b9f-47a5-933d-ad47e65bd220/volumes" Oct 30 05:39:53.160698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:53.160684 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" path="/var/lib/kubelet/pods/db2d872c-1423-440f-8e42-0bd43c2c4f07/volumes" Oct 30 05:39:53.966188 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:53.966152 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 05:39:56.821106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.821036 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:39:56.892588 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.892549 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\" (UniqueName: \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\") pod \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " Oct 30 05:39:56.892789 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.892648 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\") pod \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\" (UID: \"bdcdd147-8a32-4b22-8538-7e9ee4249c8c\") " Oct 30 05:39:56.893028 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.892996 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume" pod "bdcdd147-8a32-4b22-8538-7e9ee4249c8c" (UID: "bdcdd147-8a32-4b22-8538-7e9ee4249c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:56.893146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.893085 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:56.894739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.894716 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2" pod "bdcdd147-8a32-4b22-8538-7e9ee4249c8c" (UID: "bdcdd147-8a32-4b22-8538-7e9ee4249c8c"). InnerVolumeSpecName "kube-api-access-x65k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:56.994671 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:56.994635 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65k2\" (UniqueName: \"kubernetes.io/projected/bdcdd147-8a32-4b22-8538-7e9ee4249c8c-kube-api-access-x65k2\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:57.582039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.581945 5058 generic.go:356] "Generic (PLEG): container finished" podID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerID="98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011" exitCode=0 Oct 30 05:39:57.582039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.581994 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" Oct 30 05:39:57.582039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.582024 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" event={"ID":"bdcdd147-8a32-4b22-8538-7e9ee4249c8c","Type":"ContainerDied","Data":"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011"} Oct 30 05:39:57.582328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.582085 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs" event={"ID":"bdcdd147-8a32-4b22-8538-7e9ee4249c8c","Type":"ContainerDied","Data":"c64e7a311ab951871845f2ec992ca249a829a104c77c1c3c5d03e2bfda26b8a7"} Oct 30 05:39:57.582328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.582107 5058 scope.go:122] "RemoveContainer" containerID="98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011" Oct 30 05:39:57.591422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.591400 5058 scope.go:122] "RemoveContainer" containerID="98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011" Oct 30 05:39:57.591696 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:57.591676 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011\": container with ID starting with 98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011 not found: ID does not exist" containerID="98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011" Oct 30 05:39:57.591747 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.591709 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011"} err="failed to get container status \"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011\": rpc error: code = NotFound desc = could not find container \"98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011\": container with ID starting with 98def4bd87dc71f546167158830c4ef4c08e95aea6e9d968b27b0b1527208011 not found: ID does not exist" Oct 30 05:39:57.649888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.649853 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:39:57.663058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:57.663024 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs"] Oct 30 05:39:58.966692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:58.966625 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" podUID="7a9c17ec-b328-4605-8f42-b85013ce2798" containerName="payload-pre-processing" containerID="cri-o://36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e" gracePeriod=25 Oct 30 05:39:59.161528 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.161483 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" path="/var/lib/kubelet/pods/bdcdd147-8a32-4b22-8538-7e9ee4249c8c/volumes" Oct 30 05:39:59.225229 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.225155 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 05:39:59.321638 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.321601 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\" (UniqueName: \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\") pod \"7a9c17ec-b328-4605-8f42-b85013ce2798\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " Oct 30 05:39:59.321816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.321798 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\") pod \"7a9c17ec-b328-4605-8f42-b85013ce2798\" (UID: \"7a9c17ec-b328-4605-8f42-b85013ce2798\") " Oct 30 05:39:59.322192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.322160 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume" pod "7a9c17ec-b328-4605-8f42-b85013ce2798" (UID: "7a9c17ec-b328-4605-8f42-b85013ce2798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:59.323839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.323813 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk" pod "7a9c17ec-b328-4605-8f42-b85013ce2798" (UID: "7a9c17ec-b328-4605-8f42-b85013ce2798"). InnerVolumeSpecName "kube-api-access-x8tnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:59.423718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.423687 5058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c17ec-b328-4605-8f42-b85013ce2798-plugins-config-volume\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:59.423718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.423717 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8tnk\" (UniqueName: \"kubernetes.io/projected/7a9c17ec-b328-4605-8f42-b85013ce2798-kube-api-access-x8tnk\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:39:59.595139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.595048 5058 generic.go:356] "Generic (PLEG): container finished" podID="7a9c17ec-b328-4605-8f42-b85013ce2798" containerID="36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e" exitCode=0 Oct 30 05:39:59.595139 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.595096 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" Oct 30 05:39:59.595435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.595129 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" event={"ID":"7a9c17ec-b328-4605-8f42-b85013ce2798","Type":"ContainerDied","Data":"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e"} Oct 30 05:39:59.595435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.595199 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k" event={"ID":"7a9c17ec-b328-4605-8f42-b85013ce2798","Type":"ContainerDied","Data":"43c64dae2dad37485e644657eca41c2c922084c032db0d9d9061132eacc21156"} Oct 30 05:39:59.595435 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.595250 5058 scope.go:122] "RemoveContainer" containerID="36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e" Oct 30 05:39:59.609857 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.609838 5058 scope.go:122] "RemoveContainer" containerID="36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e" Oct 30 05:39:59.610112 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:39:59.610093 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e\": container with ID starting with 36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e not found: ID does not exist" containerID="36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e" Oct 30 05:39:59.610155 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.610121 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e"} err="failed to get container status \"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e\": rpc error: code = NotFound desc = could not find container \"36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e\": container with ID starting with 36c60c1937129dd620a16fc8056d55bbece123f3459f7245a6cf81511208212e not found: ID does not exist" Oct 30 05:39:59.657721 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.657686 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 05:39:59.676424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:39:59.676392 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k"] Oct 30 05:40:01.159838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.159803 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9c17ec-b328-4605-8f42-b85013ce2798" path="/var/lib/kubelet/pods/7a9c17ec-b328-4605-8f42-b85013ce2798/volumes" Oct 30 05:40:01.455058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.455024 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:40:01.455373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.455342 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" podUID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" containerName="maas-api" containerID="cri-o://57a13ebd5b6d3027bce4819129e74e8fbcdb611b38319fcc5be2279597423d5b" gracePeriod=30 Oct 30 05:40:01.609737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.609704 5058 generic.go:356] "Generic (PLEG): container finished" podID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" containerID="57a13ebd5b6d3027bce4819129e74e8fbcdb611b38319fcc5be2279597423d5b" exitCode=0 Oct 30 05:40:01.609904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.609791 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" event={"ID":"33b9638d-bef8-40f5-88b7-3eca979ad2a5","Type":"ContainerDied","Data":"57a13ebd5b6d3027bce4819129e74e8fbcdb611b38319fcc5be2279597423d5b"} Oct 30 05:40:01.721080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.721001 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:40:01.848612 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.848580 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\" (UniqueName: \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\") pod \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " Oct 30 05:40:01.848818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.848677 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") pod \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " Oct 30 05:40:01.848818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.848755 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\") pod \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " Oct 30 05:40:01.848818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.848776 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\") pod \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\" (UID: \"33b9638d-bef8-40f5-88b7-3eca979ad2a5\") " Oct 30 05:40:01.849179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.849148 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca" pod "33b9638d-bef8-40f5-88b7-3eca979ad2a5" (UID: "33b9638d-bef8-40f5-88b7-3eca979ad2a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:01.850800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.850766 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t" pod "33b9638d-bef8-40f5-88b7-3eca979ad2a5" (UID: "33b9638d-bef8-40f5-88b7-3eca979ad2a5"). InnerVolumeSpecName "kube-api-access-2tl6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:01.850904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.850808 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls" pod "33b9638d-bef8-40f5-88b7-3eca979ad2a5" (UID: "33b9638d-bef8-40f5-88b7-3eca979ad2a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:01.850904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.850856 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs" pod "33b9638d-bef8-40f5-88b7-3eca979ad2a5" (UID: "33b9638d-bef8-40f5-88b7-3eca979ad2a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:01.950615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.950569 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tl6t\" (UniqueName: \"kubernetes.io/projected/33b9638d-bef8-40f5-88b7-3eca979ad2a5-kube-api-access-2tl6t\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:01.950615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.950606 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:01.950615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.950620 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b9638d-bef8-40f5-88b7-3eca979ad2a5-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:01.950866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:01.950631 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b9638d-bef8-40f5-88b7-3eca979ad2a5-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:02.617072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:02.617045 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" Oct 30 05:40:02.617072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:02.617059 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt" event={"ID":"33b9638d-bef8-40f5-88b7-3eca979ad2a5","Type":"ContainerDied","Data":"45fa39a588cc305cc129e2a85d803c39714c2c38a5c3281c54c0383a340f0dae"} Oct 30 05:40:02.617622 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:02.617113 5058 scope.go:122] "RemoveContainer" containerID="57a13ebd5b6d3027bce4819129e74e8fbcdb611b38319fcc5be2279597423d5b" Oct 30 05:40:02.674773 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:02.674739 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:40:02.688485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:02.688455 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9dc17a-6954fd69b4-rk7qt"] Oct 30 05:40:03.160169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:03.160133 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" path="/var/lib/kubelet/pods/33b9638d-bef8-40f5-88b7-3eca979ad2a5/volumes" Oct 30 05:40:09.834431 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:09.834397 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:40:09.834851 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:09.834585 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" containerID="cri-o://9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb" gracePeriod=30 Oct 30 05:40:13.130794 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.130758 5058 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:40:13.131369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.130814 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:40:13.224091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.224031 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 05:40:13.224360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.224311 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" podUID="aeb17cf5-c745-487d-b7cc-747255962479" containerName="maas-api" containerID="cri-o://1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819" gracePeriod=30 Oct 30 05:40:13.486567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.486532 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 05:40:13.580311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580275 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\") pod \"aeb17cf5-c745-487d-b7cc-747255962479\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " Oct 30 05:40:13.580480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580375 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\") pod \"aeb17cf5-c745-487d-b7cc-747255962479\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " Oct 30 05:40:13.580529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580503 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\" (UniqueName: \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\") pod \"aeb17cf5-c745-487d-b7cc-747255962479\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " Oct 30 05:40:13.580579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580565 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") pod \"aeb17cf5-c745-487d-b7cc-747255962479\" (UID: \"aeb17cf5-c745-487d-b7cc-747255962479\") " Oct 30 05:40:13.580695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580669 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca" pod "aeb17cf5-c745-487d-b7cc-747255962479" (UID: "aeb17cf5-c745-487d-b7cc-747255962479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:13.580960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.580935 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb17cf5-c745-487d-b7cc-747255962479-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.582770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.582734 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls" pod "aeb17cf5-c745-487d-b7cc-747255962479" (UID: "aeb17cf5-c745-487d-b7cc-747255962479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:13.582961 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.582943 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm" pod "aeb17cf5-c745-487d-b7cc-747255962479" (UID: "aeb17cf5-c745-487d-b7cc-747255962479"). InnerVolumeSpecName "kube-api-access-gcfvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:13.583012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.582945 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs" pod "aeb17cf5-c745-487d-b7cc-747255962479" (UID: "aeb17cf5-c745-487d-b7cc-747255962479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.680809 5058 generic.go:356] "Generic (PLEG): container finished" podID="aeb17cf5-c745-487d-b7cc-747255962479" containerID="1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819" exitCode=0 Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.680870 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.680871 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" event={"ID":"aeb17cf5-c745-487d-b7cc-747255962479","Type":"ContainerDied","Data":"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819"} Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.680957 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw" event={"ID":"aeb17cf5-c745-487d-b7cc-747255962479","Type":"ContainerDied","Data":"545bde26b36fe33e54fc76e4b7bf536da9c7b153dc6a98df738fa041eb00efa0"} Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.680993 5058 scope.go:122] "RemoveContainer" containerID="1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819" Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.681654 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfvm\" (UniqueName: \"kubernetes.io/projected/aeb17cf5-c745-487d-b7cc-747255962479-kube-api-access-gcfvm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.681677 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.682584 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.681692 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb17cf5-c745-487d-b7cc-747255962479-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.691472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.691366 5058 scope.go:122] "RemoveContainer" containerID="1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819" Oct 30 05:40:13.691834 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:13.691786 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819\": container with ID starting with 1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819 not found: ID does not exist" containerID="1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819" Oct 30 05:40:13.691922 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.691823 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819"} err="failed to get container status \"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819\": rpc error: code = NotFound desc = could not find container \"1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819\": container with ID starting with 1ace9933ad755787450a03d6b2f1bce9c58c19958137df8a19e7cf393f6f9819 not found: ID does not exist" Oct 30 05:40:13.739154 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.739119 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 05:40:13.741053 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:13.741030 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0b21ba-878558655-s4bsw"] Oct 30 05:40:15.109274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.109198 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:40:15.163316 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.163277 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb17cf5-c745-487d-b7cc-747255962479" path="/var/lib/kubelet/pods/aeb17cf5-c745-487d-b7cc-747255962479/volumes" Oct 30 05:40:15.199350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199317 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199355 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199436 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199471 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199503 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199534 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199588 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199594 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199645 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199674 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199692 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\") pod \"785b3321-964e-48f9-804b-bb1b7c76b9dd\" (UID: \"785b3321-964e-48f9-804b-bb1b7c76b9dd\") " Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199800 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.199927 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199914 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.199934 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.200085 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785b3321-964e-48f9-804b-bb1b7c76b9dd-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.200106 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.200119 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.200133 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.200360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.200150 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.201852 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.201829 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.201972 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.201919 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "kube-api-access-dlqfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.202318 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.202284 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:15.202547 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.202517 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token" pod "785b3321-964e-48f9-804b-bb1b7c76b9dd" (UID: "785b3321-964e-48f9-804b-bb1b7c76b9dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.300899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.300862 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.300899 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.300901 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.301091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.300916 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785b3321-964e-48f9-804b-bb1b7c76b9dd-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.301091 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.300930 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqfb\" (UniqueName: \"kubernetes.io/projected/785b3321-964e-48f9-804b-bb1b7c76b9dd-kube-api-access-dlqfb\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.336981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.336943 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 05:40:15.343836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.343799 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 05:40:15.694468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.694419 5058 generic.go:356] "Generic (PLEG): container finished" podID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerID="9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb" exitCode=0 Oct 30 05:40:15.694468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.694468 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" Oct 30 05:40:15.694725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.694506 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" event={"ID":"785b3321-964e-48f9-804b-bb1b7c76b9dd","Type":"ContainerDied","Data":"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb"} Oct 30 05:40:15.694725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.694568 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr" event={"ID":"785b3321-964e-48f9-804b-bb1b7c76b9dd","Type":"ContainerDied","Data":"108a675daa53c630af69f2c7f0489a23e6c4b605487a98918637d57273716f64"} Oct 30 05:40:15.694725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.694600 5058 scope.go:122] "RemoveContainer" containerID="9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb" Oct 30 05:40:15.705099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.705074 5058 scope.go:122] "RemoveContainer" containerID="9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb" Oct 30 05:40:15.705464 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:15.705435 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb\": container with ID starting with 9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb not found: ID does not exist" containerID="9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb" Oct 30 05:40:15.705606 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.705476 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb"} err="failed to get container status \"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb\": rpc error: code = NotFound desc = could not find container \"9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb\": container with ID starting with 9d8aceecd72a535659c6eec8b252b36cf0b326530e5a56f7983d9075b5016adb not found: ID does not exist" Oct 30 05:40:15.778957 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.778924 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:40:15.815943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:15.815912 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr"] Oct 30 05:40:17.161831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:17.161794 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" path="/var/lib/kubelet/pods/785b3321-964e-48f9-804b-bb1b7c76b9dd/volumes" Oct 30 05:40:19.188551 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:19.188513 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:40:30.358299 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.358260 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" containerID="cri-o://a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb" gracePeriod=45 Oct 30 05:40:30.365663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.365610 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" containerID="cri-o://e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8" gracePeriod=45 Oct 30 05:40:30.629708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.629683 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:40:30.644006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.643981 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:40:30.759270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759231 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.759439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759306 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.759439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759373 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.759439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759401 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759442 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759489 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759514 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759536 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759542 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759564 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\" (UniqueName: \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.759627 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759600 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\" (UniqueName: \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759651 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759693 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\") pod \"35f1263b-5f14-478a-9174-d7798ce3a59d\" (UID: \"35f1263b-5f14-478a-9174-d7798ce3a59d\") " Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759710 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759709 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759745 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\" (UniqueName: \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\") pod \"833a27e7-688a-4148-a8ef-71cb6cf67cac\" (UID: \"833a27e7-688a-4148-a8ef-71cb6cf67cac\") " Oct 30 05:40:30.760018 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.759933 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.760396 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.760089 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.762118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.760991 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.762118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.761021 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.762118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.761035 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.762118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.761050 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.762118 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.761068 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.771423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771387 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.771423 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771399 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:30.771683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771387 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:30.771683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771441 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.771683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771464 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7" pod "833a27e7-688a-4148-a8ef-71cb6cf67cac" (UID: "833a27e7-688a-4148-a8ef-71cb6cf67cac"). InnerVolumeSpecName "kube-api-access-98fz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:30.771683 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.771499 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln" pod "35f1263b-5f14-478a-9174-d7798ce3a59d" (UID: "35f1263b-5f14-478a-9174-d7798ce3a59d"). InnerVolumeSpecName "kube-api-access-ggvln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:30.783127 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.783094 5058 generic.go:356] "Generic (PLEG): container finished" podID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerID="a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb" exitCode=0 Oct 30 05:40:30.783320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.783151 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" Oct 30 05:40:30.783320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.783174 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" event={"ID":"35f1263b-5f14-478a-9174-d7798ce3a59d","Type":"ContainerDied","Data":"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb"} Oct 30 05:40:30.783320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.783245 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5" event={"ID":"35f1263b-5f14-478a-9174-d7798ce3a59d","Type":"ContainerDied","Data":"5e5a7706e0aff87f2be7f42eb63a99c75994247f21a4466b4c4d1f3ba6c5a314"} Oct 30 05:40:30.783320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.783265 5058 scope.go:122] "RemoveContainer" containerID="a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb" Oct 30 05:40:30.784846 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.784807 5058 generic.go:356] "Generic (PLEG): container finished" podID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerID="e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8" exitCode=0 Oct 30 05:40:30.784960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.784851 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" event={"ID":"833a27e7-688a-4148-a8ef-71cb6cf67cac","Type":"ContainerDied","Data":"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8"} Oct 30 05:40:30.784960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.784898 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" event={"ID":"833a27e7-688a-4148-a8ef-71cb6cf67cac","Type":"ContainerDied","Data":"8da8c1392af520302e73a72ecbc7b609ebf09c4cf90870cc6712fd57aee491cf"} Oct 30 05:40:30.784960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.784925 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb" Oct 30 05:40:30.793840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.793746 5058 scope.go:122] "RemoveContainer" containerID="a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb" Oct 30 05:40:30.794070 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:30.794049 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb\": container with ID starting with a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb not found: ID does not exist" containerID="a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb" Oct 30 05:40:30.794151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.794077 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb"} err="failed to get container status \"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb\": rpc error: code = NotFound desc = could not find container \"a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb\": container with ID starting with a3463305785cf316c9d7b3ab67f83405500699be15f6c71feb19b89eb13116fb not found: ID does not exist" Oct 30 05:40:30.794151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.794097 5058 scope.go:122] "RemoveContainer" containerID="e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8" Oct 30 05:40:30.802619 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.802589 5058 scope.go:122] "RemoveContainer" containerID="e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8" Oct 30 05:40:30.802901 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:30.802880 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8\": container with ID starting with e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8 not found: ID does not exist" containerID="e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8" Oct 30 05:40:30.802965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.802912 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8"} err="failed to get container status \"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8\": rpc error: code = NotFound desc = could not find container \"e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8\": container with ID starting with e50526aa77581f16978e8780e188a7ef1919099917d5ec0f18b94acac20c70e8 not found: ID does not exist" Oct 30 05:40:30.840525 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.840499 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 05:40:30.842132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.842112 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-2-w1-cff4ac-kserve-6d959867b-w77t5"] Oct 30 05:40:30.862142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862117 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862140 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/35f1263b-5f14-478a-9174-d7798ce3a59d-kube-api-access-ggvln\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862152 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f1263b-5f14-478a-9174-d7798ce3a59d-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862163 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833a27e7-688a-4148-a8ef-71cb6cf67cac-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862170 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f1263b-5f14-478a-9174-d7798ce3a59d-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862178 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98fz7\" (UniqueName: \"kubernetes.io/projected/833a27e7-688a-4148-a8ef-71cb6cf67cac-kube-api-access-98fz7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.862336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.862187 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833a27e7-688a-4148-a8ef-71cb6cf67cac-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.879603 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.879527 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 05:40:30.881642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:30.881617 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/distinct-w1-cff4ac-kserve-64676d85f9-7dpkb"] Oct 30 05:40:31.159864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:31.159835 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" path="/var/lib/kubelet/pods/35f1263b-5f14-478a-9174-d7798ce3a59d/volumes" Oct 30 05:40:31.160297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:31.160282 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" path="/var/lib/kubelet/pods/833a27e7-688a-4148-a8ef-71cb6cf67cac/volumes" Oct 30 05:40:36.762268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:36.762234 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 05:40:36.762686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:36.762449 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" podUID="dc3b3b52-4661-4930-a6b1-08e0c350337b" containerName="maas-api" containerID="cri-o://06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31" gracePeriod=30 Oct 30 05:40:37.030624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.030552 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 05:40:37.122743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.122709 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\") pod \"dc3b3b52-4661-4930-a6b1-08e0c350337b\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " Oct 30 05:40:37.122929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.122800 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\") pod \"dc3b3b52-4661-4930-a6b1-08e0c350337b\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " Oct 30 05:40:37.122929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.122836 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\") pod \"dc3b3b52-4661-4930-a6b1-08e0c350337b\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " Oct 30 05:40:37.123066 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.122941 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") pod \"dc3b3b52-4661-4930-a6b1-08e0c350337b\" (UID: \"dc3b3b52-4661-4930-a6b1-08e0c350337b\") " Oct 30 05:40:37.123264 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.123199 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca" pod "dc3b3b52-4661-4930-a6b1-08e0c350337b" (UID: "dc3b3b52-4661-4930-a6b1-08e0c350337b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:37.124982 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.124958 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq" pod "dc3b3b52-4661-4930-a6b1-08e0c350337b" (UID: "dc3b3b52-4661-4930-a6b1-08e0c350337b"). InnerVolumeSpecName "kube-api-access-zftsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:37.125079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.124981 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs" pod "dc3b3b52-4661-4930-a6b1-08e0c350337b" (UID: "dc3b3b52-4661-4930-a6b1-08e0c350337b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:37.125079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.125003 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls" pod "dc3b3b52-4661-4930-a6b1-08e0c350337b" (UID: "dc3b3b52-4661-4930-a6b1-08e0c350337b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:37.224071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.224036 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/dc3b3b52-4661-4930-a6b1-08e0c350337b-kube-api-access-zftsq\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:37.224071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.224065 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3b3b52-4661-4930-a6b1-08e0c350337b-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:37.224071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.224076 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:37.224345 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.224084 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3b3b52-4661-4930-a6b1-08e0c350337b-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:37.826418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.826379 5058 generic.go:356] "Generic (PLEG): container finished" podID="dc3b3b52-4661-4930-a6b1-08e0c350337b" containerID="06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31" exitCode=0 Oct 30 05:40:37.826947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.826453 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" Oct 30 05:40:37.826947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.826486 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" event={"ID":"dc3b3b52-4661-4930-a6b1-08e0c350337b","Type":"ContainerDied","Data":"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31"} Oct 30 05:40:37.826947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.826543 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf" event={"ID":"dc3b3b52-4661-4930-a6b1-08e0c350337b","Type":"ContainerDied","Data":"ddb0db5c497b9f28084ce759ae64eb367931087f2453f759f20215940cd90fd4"} Oct 30 05:40:37.826947 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.826575 5058 scope.go:122] "RemoveContainer" containerID="06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31" Oct 30 05:40:37.836065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.836046 5058 scope.go:122] "RemoveContainer" containerID="06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31" Oct 30 05:40:37.836320 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:37.836299 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31\": container with ID starting with 06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31 not found: ID does not exist" containerID="06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31" Oct 30 05:40:37.836395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.836328 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31"} err="failed to get container status \"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31\": rpc error: code = NotFound desc = could not find container \"06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31\": container with ID starting with 06bf6451cb12932b0c1a22fbb67ba2961195e3aad86ec8f0d39c3bcaff783a31 not found: ID does not exist" Oct 30 05:40:37.882023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.881989 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 05:40:37.895778 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:37.895746 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cff4ac-6c9c849f4b-x22lf"] Oct 30 05:40:39.160558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:39.160521 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3b3b52-4661-4930-a6b1-08e0c350337b" path="/var/lib/kubelet/pods/dc3b3b52-4661-4930-a6b1-08e0c350337b/volumes" Oct 30 05:40:44.335806 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:44.335770 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 05:40:44.336242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:44.336010 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" containerID="cri-o://464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3" gracePeriod=30 Oct 30 05:40:47.295292 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:47.295255 5058 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:40:47.295704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:47.295328 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:40:49.190044 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:49.190004 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:40:49.590604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.590579 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 05:40:49.639887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.639856 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.639887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.639893 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.639925 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.639990 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640018 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640065 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640117 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640146 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640261 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\") pod \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\" (UID: \"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48\") " Oct 30 05:40:49.640579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640395 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:49.640579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640506 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:49.640579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640541 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:49.640799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640625 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:49.640887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640863 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.640955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640889 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.640955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640904 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.640955 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.640921 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.641093 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.641012 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:49.642576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.642512 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:49.642712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.642657 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "kube-api-access-lbbws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:49.642842 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.642818 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:49.643330 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.643305 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy" pod "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" (UID: "e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:49.712709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.712620 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 05:40:49.712903 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.712874 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" podUID="3661465d-75ee-4b89-8e02-5dfb791997f0" containerName="manager" containerID="cri-o://41bd7fbb49fa20e886313a521ac83f339a162dceba47ff55b76c3c317051643c" gracePeriod=30 Oct 30 05:40:49.741686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.741658 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbws\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-kube-api-access-lbbws\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.741686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.741688 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.741839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.741703 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.741839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.741712 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.741839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.741720 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:49.893695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.893656 5058 generic.go:356] "Generic (PLEG): container finished" podID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerID="464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3" exitCode=0 Oct 30 05:40:49.893921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.893902 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" Oct 30 05:40:49.895334 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.895305 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" event={"ID":"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48","Type":"ContainerDied","Data":"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3"} Oct 30 05:40:49.895534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.895516 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc" event={"ID":"e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48","Type":"ContainerDied","Data":"d499c2257a75004d5b98dba183f865eb5ad9eefb109dd077ec9643062e38a156"} Oct 30 05:40:49.895669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.895584 5058 scope.go:122] "RemoveContainer" containerID="464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3" Oct 30 05:40:49.896862 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.896834 5058 generic.go:356] "Generic (PLEG): container finished" podID="3661465d-75ee-4b89-8e02-5dfb791997f0" containerID="41bd7fbb49fa20e886313a521ac83f339a162dceba47ff55b76c3c317051643c" exitCode=0 Oct 30 05:40:49.896980 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.896914 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" event={"ID":"3661465d-75ee-4b89-8e02-5dfb791997f0","Type":"ContainerDied","Data":"41bd7fbb49fa20e886313a521ac83f339a162dceba47ff55b76c3c317051643c"} Oct 30 05:40:49.906604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.906587 5058 scope.go:122] "RemoveContainer" containerID="464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3" Oct 30 05:40:49.906948 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:40:49.906910 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3\": container with ID starting with 464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3 not found: ID does not exist" containerID="464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3" Oct 30 05:40:49.907040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.906948 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3"} err="failed to get container status \"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3\": rpc error: code = NotFound desc = could not find container \"464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3\": container with ID starting with 464cdf7dd1015f7c1c8328ab630ea19c50dce3b47a4a5421e821087639f7e6e3 not found: ID does not exist" Oct 30 05:40:49.951484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.951442 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 05:40:49.966769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.966675 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc"] Oct 30 05:40:49.983490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:49.983461 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 05:40:50.044507 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.044458 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\" (UniqueName: \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\") pod \"3661465d-75ee-4b89-8e02-5dfb791997f0\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " Oct 30 05:40:50.044708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.044514 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\") pod \"3661465d-75ee-4b89-8e02-5dfb791997f0\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " Oct 30 05:40:50.044708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.044634 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\") pod \"3661465d-75ee-4b89-8e02-5dfb791997f0\" (UID: \"3661465d-75ee-4b89-8e02-5dfb791997f0\") " Oct 30 05:40:50.047666 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.047627 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert" pod "3661465d-75ee-4b89-8e02-5dfb791997f0" (UID: "3661465d-75ee-4b89-8e02-5dfb791997f0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:50.048836 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.048798 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs" pod "3661465d-75ee-4b89-8e02-5dfb791997f0" (UID: "3661465d-75ee-4b89-8e02-5dfb791997f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:50.049328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.049281 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5" pod "3661465d-75ee-4b89-8e02-5dfb791997f0" (UID: "3661465d-75ee-4b89-8e02-5dfb791997f0"). InnerVolumeSpecName "kube-api-access-ztmc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:50.145818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.145780 5058 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-webhook-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:50.145988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.145823 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmc5\" (UniqueName: \"kubernetes.io/projected/3661465d-75ee-4b89-8e02-5dfb791997f0-kube-api-access-ztmc5\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:50.145988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.145840 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3661465d-75ee-4b89-8e02-5dfb791997f0-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:40:50.653706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.653668 5058 scope.go:122] "RemoveContainer" containerID="41bd7fbb49fa20e886313a521ac83f339a162dceba47ff55b76c3c317051643c" Oct 30 05:40:50.901516 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.901482 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" event={"ID":"3661465d-75ee-4b89-8e02-5dfb791997f0","Type":"ContainerDied","Data":"7a652bbbcc5c0727856e05156e4e6770d335466fca504657a1ddd8fe1c2da4b9"} Oct 30 05:40:50.901689 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.901517 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-8t5sz" Oct 30 05:40:50.985837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.985802 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 05:40:50.991101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:50.991064 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-8t5sz"] Oct 30 05:40:51.159520 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.159445 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3661465d-75ee-4b89-8e02-5dfb791997f0" path="/var/lib/kubelet/pods/3661465d-75ee-4b89-8e02-5dfb791997f0/volumes" Oct 30 05:40:51.159888 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.159874 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" path="/var/lib/kubelet/pods/e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48/volumes" Oct 30 05:40:51.630540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630458 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:40:51.630873 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630861 5058 cpu_manager.go:414] "containerMap: removing container" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630874 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630886 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3661465d-75ee-4b89-8e02-5dfb791997f0" containerName="manager" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630892 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3661465d-75ee-4b89-8e02-5dfb791997f0" containerName="manager" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630901 5058 cpu_manager.go:414] "containerMap: removing container" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630907 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630914 5058 cpu_manager.go:414] "containerMap: removing container" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerName="maas-api" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630918 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerName="maas-api" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630924 5058 cpu_manager.go:414] "containerMap: removing container" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" Oct 30 05:40:51.630926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630930 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630938 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630943 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630953 5058 cpu_manager.go:414] "containerMap: removing container" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630958 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630964 5058 cpu_manager.go:414] "containerMap: removing container" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630969 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630976 5058 cpu_manager.go:414] "containerMap: removing container" podUID="adb747f5-5af4-4af4-b359-92de31f91784" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630980 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb747f5-5af4-4af4-b359-92de31f91784" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630989 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.630993 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631006 5058 cpu_manager.go:414] "containerMap: removing container" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631010 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631017 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631021 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631027 5058 cpu_manager.go:414] "containerMap: removing container" podUID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631031 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631038 5058 cpu_manager.go:414] "containerMap: removing container" podUID="36ad3d95-f894-40d1-a077-488e4250b79c" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631042 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ad3d95-f894-40d1-a077-488e4250b79c" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631053 5058 cpu_manager.go:414] "containerMap: removing container" podUID="496f5f36-88fe-42e6-b36c-d564ccc16717" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631057 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496f5f36-88fe-42e6-b36c-d564ccc16717" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631064 5058 cpu_manager.go:414] "containerMap: removing container" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631070 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631076 5058 cpu_manager.go:414] "containerMap: removing container" podUID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" containerName="payload-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631082 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" containerName="payload-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631088 5058 cpu_manager.go:414] "containerMap: removing container" podUID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" containerName="payload-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631093 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" containerName="payload-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631101 5058 cpu_manager.go:414] "containerMap: removing container" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631105 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631111 5058 cpu_manager.go:414] "containerMap: removing container" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631115 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631122 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7753b434-1229-4fe8-8848-560519c4028a" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631127 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753b434-1229-4fe8-8848-560519c4028a" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631133 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631137 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631146 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631151 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631157 5058 cpu_manager.go:414] "containerMap: removing container" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631161 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631168 5058 cpu_manager.go:414] "containerMap: removing container" podUID="dc3b3b52-4661-4930-a6b1-08e0c350337b" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631173 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3b3b52-4661-4930-a6b1-08e0c350337b" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631178 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7a9c17ec-b328-4605-8f42-b85013ce2798" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631182 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9c17ec-b328-4605-8f42-b85013ce2798" containerName="payload-pre-processing" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631188 5058 cpu_manager.go:414] "containerMap: removing container" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631192 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631197 5058 cpu_manager.go:414] "containerMap: removing container" podUID="5cbda810-6e57-412a-a569-fbfb50859a32" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631202 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbda810-6e57-412a-a569-fbfb50859a32" containerName="maas-api" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631208 5058 cpu_manager.go:414] "containerMap: removing container" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631212 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631234 5058 cpu_manager.go:414] "containerMap: removing container" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" Oct 30 05:40:51.631205 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631239 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631246 5058 cpu_manager.go:414] "containerMap: removing container" podUID="aeb17cf5-c745-487d-b7cc-747255962479" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631251 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb17cf5-c745-487d-b7cc-747255962479" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631258 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631262 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631269 5058 cpu_manager.go:414] "containerMap: removing container" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631273 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631332 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abb0e66-ebc5-4a6a-97e6-852adf1887c0" containerName="payload-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631343 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631351 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b677a9-5b44-4b9f-a853-3d68e2ab3d94" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631359 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753b434-1229-4fe8-8848-560519c4028a" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631369 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9f62ec-4c65-45e0-8f4e-725f14779dfb" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631376 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7723d7b2-703a-45c2-9a32-a05d048c9d72" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631382 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcdd147-8a32-4b22-8538-7e9ee4249c8c" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631388 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3b3b52-4661-4930-a6b1-08e0c350337b" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631394 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9c17ec-b328-4605-8f42-b85013ce2798" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631399 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce31c438-647b-4639-b235-f2073e8e0467" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631406 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbda810-6e57-412a-a569-fbfb50859a32" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631411 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="833a27e7-688a-4148-a8ef-71cb6cf67cac" containerName="main" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631416 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b885da70-8b9f-47a5-933d-ad47e65bd220" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631421 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631426 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb17cf5-c745-487d-b7cc-747255962479" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631431 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc4ae68-d95b-4d50-9e6b-979b3bc7b04a" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631436 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60928a7-7924-46f2-a3a9-bc75a4fcc018" containerName="payload-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631441 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f1263b-5f14-478a-9174-d7798ce3a59d" containerName="main" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631450 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3661465d-75ee-4b89-8e02-5dfb791997f0" containerName="manager" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631458 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2d872c-1423-440f-8e42-0bd43c2c4f07" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631468 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c71880-4ac8-4f66-91e3-66f38ca38ea7" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631476 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="785b3321-964e-48f9-804b-bb1b7c76b9dd" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631483 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed3c3ed-05b9-4cdb-a765-f0a30cfbdf7d" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631492 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d71ee9-d4d7-4ccf-b931-cc47bdf7fb48" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631499 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="984f6054-dee9-4a26-a54c-e13180ed91b9" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631507 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb747f5-5af4-4af4-b359-92de31f91784" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631515 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631525 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03b79b2-9a5a-47cf-b430-8562ca63ceef" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631533 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b9638d-bef8-40f5-88b7-3eca979ad2a5" containerName="maas-api" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631541 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ad3d95-f894-40d1-a077-488e4250b79c" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631550 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="496f5f36-88fe-42e6-b36c-d564ccc16717" containerName="payload-pre-processing" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631559 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="945b59f2-18c2-4bfb-8050-6f0e5dc8a1d6" containerName="istio-proxy" Oct 30 05:40:51.633509 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.631566 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5308d7-6e00-4944-ab33-00c1a59a27b2" containerName="payload-processing" Oct 30 05:40:51.636671 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.636651 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.654269 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.654242 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 05:40:51.669952 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.669920 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:40:51.674451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.674424 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 05:40:51.674451 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.674424 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c5w2d\"" Oct 30 05:40:51.674658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.674454 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 05:40:51.760615 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.760579 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5stp\" (UniqueName: \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.760823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.760622 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.760823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.760653 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.862135 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.862098 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.862314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.862146 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.862314 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.862247 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5stp\" (UniqueName: \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.864697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.864669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.864803 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.864669 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.883739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.883681 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5stp\" (UniqueName: \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\") pod \"maas-controller-5d77b88d59-2jptv\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:51.949799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:51.949759 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:52.243877 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.243846 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:40:52.244725 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:40:52.244681 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6fe3ce_fc67_4b06_abba_7732848397ac.slice/crio-56e62ccb180190dbe415bbebd63e091784620cb15a838906dcd984c833f5db5f WatchSource:0}: Error finding container 56e62ccb180190dbe415bbebd63e091784620cb15a838906dcd984c833f5db5f: Status 404 returned error can't find the container with id 56e62ccb180190dbe415bbebd63e091784620cb15a838906dcd984c833f5db5f Oct 30 05:40:52.246415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.246391 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:40:52.912247 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.912196 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-2jptv" event={"ID":"9b6fe3ce-fc67-4b06-abba-7732848397ac","Type":"ContainerStarted","Data":"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24"} Oct 30 05:40:52.912658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.912269 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-2jptv" event={"ID":"9b6fe3ce-fc67-4b06-abba-7732848397ac","Type":"ContainerStarted","Data":"56e62ccb180190dbe415bbebd63e091784620cb15a838906dcd984c833f5db5f"} Oct 30 05:40:52.912658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.912306 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:40:52.959582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:40:52.959529 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d77b88d59-2jptv" podStartSLOduration=1.446053634 podStartE2EDuration="1.959514399s" totalImagesPullingTime="513.460765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:52.958957805 +0000 UTC m=+3425.928452794" watchObservedRunningTime="2026-10-30 05:40:52.959514399 +0000 UTC m=+3425.929009387" Oct 30 05:41:03.918481 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:03.918449 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:41:04.687418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.687383 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k"] Oct 30 05:41:04.687799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.687786 5058 cpu_manager.go:414] "containerMap: removing container" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:41:04.687845 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.687800 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1d7922-e2ef-44cb-9b00-51becde9339f" containerName="istio-proxy" Oct 30 05:41:04.687892 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.687884 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="61810822-ab44-48db-8060-240e7c848ca0" containerName="istio-proxy" Oct 30 05:41:04.691296 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.691279 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.720977 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.720942 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k"] Oct 30 05:41:04.781578 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781535 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszq6\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-kube-api-access-hszq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.781767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781649 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.781767 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781690 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.781914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781808 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.781914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781869 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.781914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781900 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be496eac-0d82-460a-8bda-85585235e8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.782071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781928 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.782071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.781960 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be496eac-0d82-460a-8bda-85585235e8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.782071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.782038 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.882999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.882943 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.882999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.882995 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883039 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883069 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883090 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be496eac-0d82-460a-8bda-85585235e8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883211 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883310 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be496eac-0d82-460a-8bda-85585235e8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883403 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883476 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszq6\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-kube-api-access-hszq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883786 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883769 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883523 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.883935 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883758 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.884039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.883995 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.884144 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.884115 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be496eac-0d82-460a-8bda-85585235e8ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.885535 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.885511 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be496eac-0d82-460a-8bda-85585235e8ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.885975 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.885949 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be496eac-0d82-460a-8bda-85585235e8ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.904529 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.904503 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszq6\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-kube-api-access-hszq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:04.904761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:04.904740 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be496eac-0d82-460a-8bda-85585235e8ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-68n2k\" (UID: \"be496eac-0d82-460a-8bda-85585235e8ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:05.005041 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.004957 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:05.310129 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.310099 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k"] Oct 30 05:41:05.310693 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:41:05.310670 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe496eac_0d82_460a_8bda_85585235e8ce.slice/crio-981f707667f50f240124368b5f9a84d458121078edd6fe9644176534973aedc1 WatchSource:0}: Error finding container 981f707667f50f240124368b5f9a84d458121078edd6fe9644176534973aedc1: Status 404 returned error can't find the container with id 981f707667f50f240124368b5f9a84d458121078edd6fe9644176534973aedc1 Oct 30 05:41:05.312759 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.312728 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.312850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.312790 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.312850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.312818 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.985288 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.985249 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" event={"ID":"be496eac-0d82-460a-8bda-85585235e8ce","Type":"ContainerStarted","Data":"c6ad9efa8368bcdd23c900a4f8400977cefd3f62f94e74ee0d759c7bd341e18d"} Oct 30 05:41:05.985495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:05.985308 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" event={"ID":"be496eac-0d82-460a-8bda-85585235e8ce","Type":"ContainerStarted","Data":"981f707667f50f240124368b5f9a84d458121078edd6fe9644176534973aedc1"} Oct 30 05:41:06.005712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:06.005687 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:06.007035 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:06.007013 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:06.007125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:06.007058 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:06.063049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:06.062994 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podStartSLOduration=2.062979741 podStartE2EDuration="2.062979741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:06.0614786 +0000 UTC m=+3439.030973590" watchObservedRunningTime="2026-10-30 05:41:06.062979741 +0000 UTC m=+3439.032474729" Oct 30 05:41:07.005839 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:07.005807 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:07.006266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:07.005858 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:08.005500 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:08.005464 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:08.005691 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:08.005518 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:09.006322 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:09.006288 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:09.006732 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:09.006338 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:10.005610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:10.005574 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:10.005798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:10.005627 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:11.005820 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:11.005782 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:11.006273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:11.005843 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:12.005413 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:12.005374 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 30 05:41:12.005639 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:12.005443 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 30 05:41:13.609579 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:13.609533 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:13.610391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:13.610370 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:13.611188 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:13.611167 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" Oct 30 05:41:19.189483 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:41:19.189446 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:41:26.090582 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.090547 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:41:26.091095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.090749 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5d77b88d59-2jptv" podUID="9b6fe3ce-fc67-4b06-abba-7732848397ac" containerName="manager" containerID="cri-o://f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24" gracePeriod=30 Oct 30 05:41:26.356559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.356497 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:41:26.411973 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.411935 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\") pod \"9b6fe3ce-fc67-4b06-abba-7732848397ac\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " Oct 30 05:41:26.412176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.412074 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\" (UniqueName: \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\") pod \"9b6fe3ce-fc67-4b06-abba-7732848397ac\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " Oct 30 05:41:26.412176 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.412112 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\") pod \"9b6fe3ce-fc67-4b06-abba-7732848397ac\" (UID: \"9b6fe3ce-fc67-4b06-abba-7732848397ac\") " Oct 30 05:41:26.415185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.415136 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert" pod "9b6fe3ce-fc67-4b06-abba-7732848397ac" (UID: "9b6fe3ce-fc67-4b06-abba-7732848397ac"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:41:26.415539 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.415513 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp" pod "9b6fe3ce-fc67-4b06-abba-7732848397ac" (UID: "9b6fe3ce-fc67-4b06-abba-7732848397ac"). InnerVolumeSpecName "kube-api-access-f5stp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:26.416660 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.416639 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs" pod "9b6fe3ce-fc67-4b06-abba-7732848397ac" (UID: "9b6fe3ce-fc67-4b06-abba-7732848397ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:41:26.513866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.513830 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5stp\" (UniqueName: \"kubernetes.io/projected/9b6fe3ce-fc67-4b06-abba-7732848397ac-kube-api-access-f5stp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:41:26.513866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.513864 5058 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-webhook-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:41:26.513866 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:26.513873 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6fe3ce-fc67-4b06-abba-7732848397ac-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:41:27.087188 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.087156 5058 generic.go:356] "Generic (PLEG): container finished" podID="9b6fe3ce-fc67-4b06-abba-7732848397ac" containerID="f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24" exitCode=0 Oct 30 05:41:27.087389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.087195 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-2jptv" event={"ID":"9b6fe3ce-fc67-4b06-abba-7732848397ac","Type":"ContainerDied","Data":"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24"} Oct 30 05:41:27.087389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.087205 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-2jptv" Oct 30 05:41:27.087389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.087257 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-2jptv" event={"ID":"9b6fe3ce-fc67-4b06-abba-7732848397ac","Type":"ContainerDied","Data":"56e62ccb180190dbe415bbebd63e091784620cb15a838906dcd984c833f5db5f"} Oct 30 05:41:27.087389 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.087279 5058 scope.go:122] "RemoveContainer" containerID="f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24" Oct 30 05:41:27.097134 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.096971 5058 scope.go:122] "RemoveContainer" containerID="f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24" Oct 30 05:41:27.097424 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:41:27.097306 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24\": container with ID starting with f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24 not found: ID does not exist" containerID="f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24" Oct 30 05:41:27.097424 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.097346 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24"} err="failed to get container status \"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24\": rpc error: code = NotFound desc = could not find container \"f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24\": container with ID starting with f84ff02beda0e4f59603e556de4192acad4e2fd14de6c794cdc20b14eb6b6a24 not found: ID does not exist" Oct 30 05:41:27.148799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.148764 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:41:27.162793 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:27.162752 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-2jptv"] Oct 30 05:41:28.399823 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.399737 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d77b88d59-t8d5z"] Oct 30 05:41:28.400271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.400211 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9b6fe3ce-fc67-4b06-abba-7732848397ac" containerName="manager" Oct 30 05:41:28.400271 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.400241 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6fe3ce-fc67-4b06-abba-7732848397ac" containerName="manager" Oct 30 05:41:28.400367 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.400319 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6fe3ce-fc67-4b06-abba-7732848397ac" containerName="manager" Oct 30 05:41:28.404844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.404825 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.431595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.431564 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c5w2d\"" Oct 30 05:41:28.431595 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.431579 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 05:41:28.431962 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.431943 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rtm\" (UniqueName: \"kubernetes.io/projected/8e35e726-c96c-4dae-8978-76edcaf2916f-kube-api-access-g5rtm\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.432082 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.432002 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-webhook-cert\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.432183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.432166 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-metrics-certs\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.445798 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.445769 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 05:41:28.445960 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.445856 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 05:41:28.447617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.447591 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-t8d5z"] Oct 30 05:41:28.533625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.533573 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-metrics-certs\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.533817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.533643 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rtm\" (UniqueName: \"kubernetes.io/projected/8e35e726-c96c-4dae-8978-76edcaf2916f-kube-api-access-g5rtm\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.533817 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.533697 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-webhook-cert\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.536161 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.536138 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-webhook-cert\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.536263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.536199 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e35e726-c96c-4dae-8978-76edcaf2916f-metrics-certs\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.556234 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.556191 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rtm\" (UniqueName: \"kubernetes.io/projected/8e35e726-c96c-4dae-8978-76edcaf2916f-kube-api-access-g5rtm\") pod \"maas-controller-5d77b88d59-t8d5z\" (UID: \"8e35e726-c96c-4dae-8978-76edcaf2916f\") " pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:28.718049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:28.717971 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:29.016885 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:29.016857 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77b88d59-t8d5z"] Oct 30 05:41:29.017177 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:41:29.017155 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e35e726_c96c_4dae_8978_76edcaf2916f.slice/crio-5f01666fea7b2f3d4e2bad3a1f6bed72bf4c180130af883c5c0cc11dadf21f09 WatchSource:0}: Error finding container 5f01666fea7b2f3d4e2bad3a1f6bed72bf4c180130af883c5c0cc11dadf21f09: Status 404 returned error can't find the container with id 5f01666fea7b2f3d4e2bad3a1f6bed72bf4c180130af883c5c0cc11dadf21f09 Oct 30 05:41:29.099131 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:29.099094 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" event={"ID":"8e35e726-c96c-4dae-8978-76edcaf2916f","Type":"ContainerStarted","Data":"5f01666fea7b2f3d4e2bad3a1f6bed72bf4c180130af883c5c0cc11dadf21f09"} Oct 30 05:41:29.166534 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:29.166487 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6fe3ce-fc67-4b06-abba-7732848397ac" path="/var/lib/kubelet/pods/9b6fe3ce-fc67-4b06-abba-7732848397ac/volumes" Oct 30 05:41:30.105390 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:30.105353 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" event={"ID":"8e35e726-c96c-4dae-8978-76edcaf2916f","Type":"ContainerStarted","Data":"b753518a7c4dc05baaae28a43823c83bff5e9e1964031652ead37a32181dd1b1"} Oct 30 05:41:30.105913 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:30.105467 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:30.152490 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:30.152431 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" podStartSLOduration=1.832122997 podStartE2EDuration="2.152417688s" totalImagesPullingTime="320.294691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:30.150863827 +0000 UTC m=+3463.120358837" watchObservedRunningTime="2026-10-30 05:41:30.152417688 +0000 UTC m=+3463.121912676" Oct 30 05:41:41.113260 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:41:41.113200 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d77b88d59-t8d5z" Oct 30 05:41:49.189043 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:41:49.189005 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:42:19.199105 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:42:19.199069 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:42:19.910285 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.906162 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn"] Oct 30 05:42:19.914929 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.914898 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:19.953497 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.953461 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn"] Oct 30 05:42:19.965177 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.965146 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx"] Oct 30 05:42:19.975484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.975453 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:19.998454 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:19.998424 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx"] Oct 30 05:42:20.018168 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018137 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8536d142-4135-4460-9674-02c89d76a082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018185 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018204 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018286 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8536d142-4135-4460-9674-02c89d76a082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018346 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018362 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018402 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018442 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018617 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018560 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018654 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018621 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018661 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018758 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018709 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018752 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqm7\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-kube-api-access-nbqm7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018804 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.018904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018837 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018933 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.018979 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018961 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4jj\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-kube-api-access-kv4jj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.019067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.018987 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.119650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119611 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqm7\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-kube-api-access-nbqm7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.119650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119664 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.119945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119691 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.119945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119729 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.119945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119753 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4jj\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-kube-api-access-kv4jj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.119945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119872 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.119957 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8536d142-4135-4460-9674-02c89d76a082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120043 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120143 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120143 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120212 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120270 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8536d142-4135-4460-9674-02c89d76a082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120306 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120303 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120327 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120340 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120418 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120445 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120477 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120527 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120567 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120594 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.120931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120908 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.121495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.120943 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.121495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.121189 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.121495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.121284 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8536d142-4135-4460-9674-02c89d76a082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.121495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.121464 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.121779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.121647 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.121779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.121712 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.122664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.122638 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8536d142-4135-4460-9674-02c89d76a082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.122664 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.122652 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8536d142-4135-4460-9674-02c89d76a082-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.123101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.123081 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.123365 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.123346 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.141564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.141526 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqm7\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-kube-api-access-nbqm7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.144275 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.144244 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4jj\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-kube-api-access-kv4jj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.145391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.145370 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8536d142-4135-4460-9674-02c89d76a082-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx\" (UID: \"8536d142-4135-4460-9674-02c89d76a082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.146432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.146404 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007cda5b-cdb8-4bb6-a49f-636045e9e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-wprbn\" (UID: \"007cda5b-cdb8-4bb6-a49f-636045e9e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.230339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.230246 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:20.302494 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.302461 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:20.545037 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.544944 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn"] Oct 30 05:42:20.554464 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:42:20.554434 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007cda5b_cdb8_4bb6_a49f_636045e9e4e0.slice/crio-a02eb5de1097a04fce7f43b1cfe210adac19825108895b88e866c2ffaee4c4f5 WatchSource:0}: Error finding container a02eb5de1097a04fce7f43b1cfe210adac19825108895b88e866c2ffaee4c4f5: Status 404 returned error can't find the container with id a02eb5de1097a04fce7f43b1cfe210adac19825108895b88e866c2ffaee4c4f5 Oct 30 05:42:20.557624 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.557543 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.557738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.557678 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.557738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.557722 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.638937 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:42:20.638906 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8536d142_4135_4460_9674_02c89d76a082.slice/crio-c5b3f0d2f1f7a322fe275274b23d55af532933acaba373304a66aef311588156 WatchSource:0}: Error finding container c5b3f0d2f1f7a322fe275274b23d55af532933acaba373304a66aef311588156: Status 404 returned error can't find the container with id c5b3f0d2f1f7a322fe275274b23d55af532933acaba373304a66aef311588156 Oct 30 05:42:20.639101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.639060 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx"] Oct 30 05:42:20.641211 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.641179 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.641340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.641329 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.641408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:20.641360 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:21.384452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.384417 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" event={"ID":"007cda5b-cdb8-4bb6-a49f-636045e9e4e0","Type":"ContainerStarted","Data":"edaaad34e4dc533f42e32e70758e28370aec57dd96011d49ce94c12f6c7d2741"} Oct 30 05:42:21.384989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.384474 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" event={"ID":"007cda5b-cdb8-4bb6-a49f-636045e9e4e0","Type":"ContainerStarted","Data":"a02eb5de1097a04fce7f43b1cfe210adac19825108895b88e866c2ffaee4c4f5"} Oct 30 05:42:21.386267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.386241 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" event={"ID":"8536d142-4135-4460-9674-02c89d76a082","Type":"ContainerStarted","Data":"f9f13215f74726b050f3dd816f90e0c7fdae91d0e89742e49dddd22d3e4de49b"} Oct 30 05:42:21.386386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.386285 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" event={"ID":"8536d142-4135-4460-9674-02c89d76a082","Type":"ContainerStarted","Data":"c5b3f0d2f1f7a322fe275274b23d55af532933acaba373304a66aef311588156"} Oct 30 05:42:21.508881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.508830 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podStartSLOduration=2.508816283 podStartE2EDuration="2.508816283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:21.508080607 +0000 UTC m=+3514.477575596" watchObservedRunningTime="2026-10-30 05:42:21.508816283 +0000 UTC m=+3514.478311265" Oct 30 05:42:21.509886 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:21.509857 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podStartSLOduration=2.509850213 podStartE2EDuration="2.509850213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:21.462817819 +0000 UTC m=+3514.432312841" watchObservedRunningTime="2026-10-30 05:42:21.509850213 +0000 UTC m=+3514.479345201" Oct 30 05:42:22.233242 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.232587 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:22.242241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.240077 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:22.242241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.240143 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:22.304185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.304153 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:22.305833 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.305809 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:22.305963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:22.305856 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:23.249245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:23.248321 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:23.249245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:23.248384 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:23.304017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:23.303985 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:23.304187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:23.304035 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:24.241244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:24.238928 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:24.241244 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:24.238989 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:24.305023 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:24.304986 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:24.305484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:24.305040 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:25.239686 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:25.239643 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:25.239994 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:25.239960 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:25.304020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:25.303976 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:25.304243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:25.304028 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:26.232716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:26.232674 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:26.233394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:26.233358 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:26.304207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:26.304174 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:26.304412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:26.304329 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:27.231574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:27.231533 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:42:27.231750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:27.231593 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 05:42:27.304350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:27.304311 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 05:42:27.304800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:27.304365 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 05:42:28.847373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.847341 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:28.848591 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.848556 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:28.849380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.849355 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" Oct 30 05:42:28.985629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.985575 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:28.986565 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.986494 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:28.987532 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:28.987508 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" Oct 30 05:42:42.207764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.207722 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 05:42:42.210417 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.207910 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerName="discovery" containerID="cri-o://2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c" gracePeriod=30 Oct 30 05:42:42.471628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.471566 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 05:42:42.509725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.509685 5058 generic.go:356] "Generic (PLEG): container finished" podID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerID="2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c" exitCode=0 Oct 30 05:42:42.509904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.509745 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" Oct 30 05:42:42.509904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.509754 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" event={"ID":"12d5b74b-ff1d-404f-850f-ab1f898a198f","Type":"ContainerDied","Data":"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c"} Oct 30 05:42:42.509904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.509858 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl" event={"ID":"12d5b74b-ff1d-404f-850f-ab1f898a198f","Type":"ContainerDied","Data":"9a81fdc51bbffc75d46922cf9b421d4b018a57047af513bcfacc0bf08f9b899b"} Oct 30 05:42:42.509904 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.509877 5058 scope.go:122] "RemoveContainer" containerID="2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c" Oct 30 05:42:42.528629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.528603 5058 scope.go:122] "RemoveContainer" containerID="2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c" Oct 30 05:42:42.528934 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:42:42.528916 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c\": container with ID starting with 2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c not found: ID does not exist" containerID="2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c" Oct 30 05:42:42.528990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.528944 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c"} err="failed to get container status \"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c\": rpc error: code = NotFound desc = could not find container \"2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c\": container with ID starting with 2ee7d1005ff85a1e90ea78bd8de312387edc60707d7f454484e124396972b45c not found: ID does not exist" Oct 30 05:42:42.532506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532479 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532541 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532573 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532633 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532613 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532664 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532728 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.532816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.532760 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\") pod \"12d5b74b-ff1d-404f-850f-ab1f898a198f\" (UID: \"12d5b74b-ff1d-404f-850f-ab1f898a198f\") " Oct 30 05:42:42.533375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.533163 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:42:42.535050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535027 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:42:42.535340 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535303 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "kube-api-access-xdlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:42:42.535533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535512 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:42.535607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535528 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:42.535676 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535622 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:42.535740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.535686 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs" pod "12d5b74b-ff1d-404f-850f-ab1f898a198f" (UID: "12d5b74b-ff1d-404f-850f-ab1f898a198f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:42:42.633688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633648 5058 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-cacerts\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633681 5058 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d5b74b-ff1d-404f-850f-ab1f898a198f-local-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633688 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633692 5058 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-kubeconfig\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633702 5058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-dns-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633712 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633720 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdlq7\" (UniqueName: \"kubernetes.io/projected/12d5b74b-ff1d-404f-850f-ab1f898a198f-kube-api-access-xdlq7\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.633938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.633729 5058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12d5b74b-ff1d-404f-850f-ab1f898a198f-istio-csr-ca-configmap\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:42:42.877169 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.877126 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 05:42:42.901567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:42.901535 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl"] Oct 30 05:42:43.160875 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:43.160840 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" path="/var/lib/kubelet/pods/12d5b74b-ff1d-404f-850f-ab1f898a198f/volumes" Oct 30 05:42:49.190388 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:42:49.190334 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:42:52.341366 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:52.341319 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:42:52.342067 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:42:52.342030 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:43:19.189613 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:43:19.189569 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:43:47.920860 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:43:47.920827 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:43:47.936537 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:43:47.936492 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:43:49.205471 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:43:49.205419 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:44:19.189841 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:44:19.189797 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:44:29.661263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:29.660349 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:29.661263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:29.660445 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:29.894243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:29.893442 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:29.894243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:29.893515 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:30.059480 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:30.059358 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:30.059838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:44:30.059804 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:49.189204 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:44:49.189160 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:45:19.190098 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:45:19.190051 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:45:42.252328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.252291 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:45:42.252981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.252956 5058 cpu_manager.go:414] "containerMap: removing container" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerName="discovery" Oct 30 05:45:42.253089 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.252981 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerName="discovery" Oct 30 05:45:42.253150 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.253104 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d5b74b-ff1d-404f-850f-ab1f898a198f" containerName="discovery" Oct 30 05:45:42.256353 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.256333 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.274872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.274841 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f97a49-openshift-default-dockercfg-nxdzn\"" Oct 30 05:45:42.290446 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.290414 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:45:42.297882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.297848 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcxz\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.297882 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.297886 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.297904 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298011 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298075 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298104 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298171 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298265 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.298305 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.298288 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399355 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399393 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399401 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399514 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399564 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399588 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399650 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.399681 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399670 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399750 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcxz\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399786 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.399924 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.400016 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.400021 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.400546 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.400568 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.400562 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.402254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.402233 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.402459 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.402436 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.420694 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.420663 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcxz\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.420819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.420749 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\") pod \"e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.570831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.570731 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:42.881491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.881453 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:45:42.881917 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:45:42.881893 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a7beec_f194_4a62_80f8_89005d2e1d83.slice/crio-de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2 WatchSource:0}: Error finding container de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2: Status 404 returned error can't find the container with id de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2 Oct 30 05:45:42.884038 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.884009 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:42.884123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.884070 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:42.884123 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:42.884097 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:43.677730 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:43.677693 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerStarted","Data":"2b605319ac24d037f4b32dd991009f95feda704106d7278790947f49dc7ee82b"} Oct 30 05:45:43.678259 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:43.677742 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerStarted","Data":"de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2"} Oct 30 05:45:43.736442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:43.736398 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podStartSLOduration=1.736385781 podStartE2EDuration="1.736385781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:43.735897722 +0000 UTC m=+3716.705392711" watchObservedRunningTime="2026-10-30 05:45:43.736385781 +0000 UTC m=+3716.705880770" Oct 30 05:45:44.571743 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:44.571711 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:44.576714 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:44.576688 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:44.683444 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:44.683407 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:44.684476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:44.684436 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:45:47.611462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.611425 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:45:47.615420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.615386 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.653678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.643594 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f97a49-serving-cert\"" Oct 30 05:45:47.653678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.649708 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg9b\" (UniqueName: \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.653678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.649742 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.653678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.649830 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.653678 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.649882 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.660566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.660534 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:45:47.750613 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.750577 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.750799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.750636 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg9b\" (UniqueName: \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.750799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.750655 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.750799 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.750692 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.751432 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.751403 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.753104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.753073 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.753320 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.753299 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.771149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.771114 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg9b\" (UniqueName: \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\") pod \"maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:47.952917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:47.952888 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:48.252399 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:48.252369 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:45:48.252923 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:45:48.252894 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda244ebdd_7ded_433f_a5a8_19d7a4fd9532.slice/crio-7d30ff8b3f4feabfa1e73e81fd4b994cc024b56bc837a965cb3453f1d3148f4c WatchSource:0}: Error finding container 7d30ff8b3f4feabfa1e73e81fd4b994cc024b56bc837a965cb3453f1d3148f4c: Status 404 returned error can't find the container with id 7d30ff8b3f4feabfa1e73e81fd4b994cc024b56bc837a965cb3453f1d3148f4c Oct 30 05:45:48.705695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:48.705658 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" event={"ID":"a244ebdd-7ded-433f-a5a8-19d7a4fd9532","Type":"ContainerStarted","Data":"6dfe12554c5a6a8dd83452d81f95789e6d7ef4dc18145d401eb4fa8122c17376"} Oct 30 05:45:48.706071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:48.705722 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:45:48.706071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:48.705740 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" event={"ID":"a244ebdd-7ded-433f-a5a8-19d7a4fd9532","Type":"ContainerStarted","Data":"7d30ff8b3f4feabfa1e73e81fd4b994cc024b56bc837a965cb3453f1d3148f4c"} Oct 30 05:45:48.806272 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:48.806194 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" podStartSLOduration=1.450202421 podStartE2EDuration="1.806178975s" totalImagesPullingTime="355.976554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:48.805609882 +0000 UTC m=+3721.775104870" watchObservedRunningTime="2026-10-30 05:45:48.806178975 +0000 UTC m=+3721.775673954" Oct 30 05:45:49.190391 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:45:49.190354 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:45:54.716563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:45:54.716529 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:46:00.725391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:00.725339 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:46:00.725901 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:00.725419 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:46:01.803856 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.803822 5058 generic.go:356] "Generic (PLEG): container finished" podID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerID="2b605319ac24d037f4b32dd991009f95feda704106d7278790947f49dc7ee82b" exitCode=0 Oct 30 05:46:01.804348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.803865 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerDied","Data":"2b605319ac24d037f4b32dd991009f95feda704106d7278790947f49dc7ee82b"} Oct 30 05:46:01.804348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.804258 5058 scope.go:122] "RemoveContainer" containerID="2b605319ac24d037f4b32dd991009f95feda704106d7278790947f49dc7ee82b" Oct 30 05:46:01.805416 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.805387 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:01.805515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.805453 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:01.805515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:01.805495 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:02.810736 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:02.810701 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerStarted","Data":"8faec6a090f228a4bc0152642242634f75a8e18de3bcbf90e5c4d6e1cc26d4e0"} Oct 30 05:46:03.571706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:03.571667 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:46:03.572000 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:03.571976 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:46:03.572062 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:03.572027 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 05:46:04.195147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.195103 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:04.199270 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.199248 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.256501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.248774 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f97a49\"/\"kube-root-ca.crt\"" Oct 30 05:46:04.256501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.251450 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:04.256501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.251592 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f97a49\"/\"openshift-service-ca.crt\"" Oct 30 05:46:04.264391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264358 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.264540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264410 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.264540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264489 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.264540 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264525 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.264655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264601 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.264699 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.264679 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.268822 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.268792 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f97a49\"/\"default-dockercfg-z2v88\"" Oct 30 05:46:04.268964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.268888 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f97a49\"/\"simulator-single-main-f97a49-kserve-self-signed-certs\"" Oct 30 05:46:04.365350 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365310 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365375 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365423 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365461 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365567 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365531 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365574 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365818 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365795 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.365931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.365855 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.366080 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.366060 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.367553 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.367530 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.367986 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.367968 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.388608 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.388582 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\") pod \"simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.549526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.549415 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:04.571611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.571583 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:46:04.571761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.571629 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 05:46:04.854293 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:04.854259 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:04.854831 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:46:04.854806 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod979cc233_3759_4e11_8158_21ac6677ebd8.slice/crio-c1b4b8e10de538c2524ff78a2395a46e0ef3bb23e774b9e84029d87051cfa54b WatchSource:0}: Error finding container c1b4b8e10de538c2524ff78a2395a46e0ef3bb23e774b9e84029d87051cfa54b: Status 404 returned error can't find the container with id c1b4b8e10de538c2524ff78a2395a46e0ef3bb23e774b9e84029d87051cfa54b Oct 30 05:46:05.572031 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:05.571996 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:46:05.572510 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:05.572058 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 05:46:05.851307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:05.851184 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" event={"ID":"979cc233-3759-4e11-8158-21ac6677ebd8","Type":"ContainerStarted","Data":"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e"} Oct 30 05:46:05.851307 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:05.851262 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" event={"ID":"979cc233-3759-4e11-8158-21ac6677ebd8","Type":"ContainerStarted","Data":"c1b4b8e10de538c2524ff78a2395a46e0ef3bb23e774b9e84029d87051cfa54b"} Oct 30 05:46:05.897411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:05.897352 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" podStartSLOduration=1.897335231 podStartE2EDuration="1.897335231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:05.895349634 +0000 UTC m=+3738.864844625" watchObservedRunningTime="2026-10-30 05:46:05.897335231 +0000 UTC m=+3738.866830225" Oct 30 05:46:06.550364 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:06.550320 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:06.571095 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:06.571054 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:46:06.571287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:06.571114 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 05:46:07.571963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:07.571922 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:46:07.572405 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:07.571982 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 05:46:08.575328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:08.575296 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:46:08.575739 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:08.575554 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:46:08.576348 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:08.576324 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:46:14.550411 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:14.550377 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:14.563287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:14.563264 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:14.918792 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:14.918764 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:19.190206 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:46:19.190168 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:46:43.321119 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:43.321081 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:49.190554 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:46:49.190511 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:46:49.609290 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:49.609182 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:46:49.609496 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:49.609458 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerName="maas-api" containerID="cri-o://6dfe12554c5a6a8dd83452d81f95789e6d7ef4dc18145d401eb4fa8122c17376" gracePeriod=30 Oct 30 05:46:49.712026 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:49.711987 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.7:8443/health\": dial tcp 10.132.1.7:8443: connect: connection refused" Oct 30 05:46:50.085941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.085906 5058 generic.go:356] "Generic (PLEG): container finished" podID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerID="6dfe12554c5a6a8dd83452d81f95789e6d7ef4dc18145d401eb4fa8122c17376" exitCode=0 Oct 30 05:46:50.086094 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.085980 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" event={"ID":"a244ebdd-7ded-433f-a5a8-19d7a4fd9532","Type":"ContainerDied","Data":"6dfe12554c5a6a8dd83452d81f95789e6d7ef4dc18145d401eb4fa8122c17376"} Oct 30 05:46:50.104246 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.104204 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:46:50.141690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.141603 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\") pod \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " Oct 30 05:46:50.141864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.141712 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\") pod \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " Oct 30 05:46:50.141864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.141793 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\") pod \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " Oct 30 05:46:50.141864 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.141829 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\" (UniqueName: \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\") pod \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\" (UID: \"a244ebdd-7ded-433f-a5a8-19d7a4fd9532\") " Oct 30 05:46:50.142336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.142148 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca" pod "a244ebdd-7ded-433f-a5a8-19d7a4fd9532" (UID: "a244ebdd-7ded-433f-a5a8-19d7a4fd9532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:50.142336 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.142308 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:50.143850 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.143822 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs" pod "a244ebdd-7ded-433f-a5a8-19d7a4fd9532" (UID: "a244ebdd-7ded-433f-a5a8-19d7a4fd9532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:50.143990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.143878 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b" pod "a244ebdd-7ded-433f-a5a8-19d7a4fd9532" (UID: "a244ebdd-7ded-433f-a5a8-19d7a4fd9532"). InnerVolumeSpecName "kube-api-access-8xg9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:50.144106 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.144081 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls" pod "a244ebdd-7ded-433f-a5a8-19d7a4fd9532" (UID: "a244ebdd-7ded-433f-a5a8-19d7a4fd9532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:50.243122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.243087 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:50.243122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.243115 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:50.243122 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:50.243125 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xg9b\" (UniqueName: \"kubernetes.io/projected/a244ebdd-7ded-433f-a5a8-19d7a4fd9532-kube-api-access-8xg9b\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:51.092807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:51.092773 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" Oct 30 05:46:51.093007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:51.092809 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8" event={"ID":"a244ebdd-7ded-433f-a5a8-19d7a4fd9532","Type":"ContainerDied","Data":"7d30ff8b3f4feabfa1e73e81fd4b994cc024b56bc837a965cb3453f1d3148f4c"} Oct 30 05:46:51.093007 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:51.092873 5058 scope.go:122] "RemoveContainer" containerID="6dfe12554c5a6a8dd83452d81f95789e6d7ef4dc18145d401eb4fa8122c17376" Oct 30 05:46:51.148482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:51.148453 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:46:51.162142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:51.162118 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f97a49-5654f6655-2rlh8"] Oct 30 05:46:53.159670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:53.159637 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" path="/var/lib/kubelet/pods/a244ebdd-7ded-433f-a5a8-19d7a4fd9532/volumes" Oct 30 05:46:58.342504 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.342463 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" podUID="979cc233-3759-4e11-8158-21ac6677ebd8" containerName="main" containerID="cri-o://21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e" gracePeriod=45 Oct 30 05:46:58.599258 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.599185 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:58.626410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626376 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626414 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626443 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626482 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626538 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626667 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\") pod \"979cc233-3759-4e11-8158-21ac6677ebd8\" (UID: \"979cc233-3759-4e11-8158-21ac6677ebd8\") " Oct 30 05:46:58.626821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626713 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:46:58.626821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626726 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:46:58.627017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.626999 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:58.627083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.627017 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:58.627083 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.627023 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:46:58.628966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.628940 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "kube-api-access-9mc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:58.629099 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.629077 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:58.629146 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.629087 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm" pod "979cc233-3759-4e11-8158-21ac6677ebd8" (UID: "979cc233-3759-4e11-8158-21ac6677ebd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:46:58.728339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.728306 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:58.728339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.728332 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979cc233-3759-4e11-8158-21ac6677ebd8-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:58.728339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.728342 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979cc233-3759-4e11-8158-21ac6677ebd8-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:58.728339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:58.728351 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mc9w\" (UniqueName: \"kubernetes.io/projected/979cc233-3759-4e11-8158-21ac6677ebd8-kube-api-access-9mc9w\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:46:59.134905 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.134869 5058 generic.go:356] "Generic (PLEG): container finished" podID="979cc233-3759-4e11-8158-21ac6677ebd8" containerID="21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e" exitCode=0 Oct 30 05:46:59.135130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.134918 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" Oct 30 05:46:59.135130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.134963 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" event={"ID":"979cc233-3759-4e11-8158-21ac6677ebd8","Type":"ContainerDied","Data":"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e"} Oct 30 05:46:59.135130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.135018 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4" event={"ID":"979cc233-3759-4e11-8158-21ac6677ebd8","Type":"ContainerDied","Data":"c1b4b8e10de538c2524ff78a2395a46e0ef3bb23e774b9e84029d87051cfa54b"} Oct 30 05:46:59.135130 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.135040 5058 scope.go:122] "RemoveContainer" containerID="21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e" Oct 30 05:46:59.145327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.145310 5058 scope.go:122] "RemoveContainer" containerID="21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e" Oct 30 05:46:59.145594 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:46:59.145573 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e\": container with ID starting with 21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e not found: ID does not exist" containerID="21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e" Oct 30 05:46:59.145652 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.145604 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e"} err="failed to get container status \"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e\": rpc error: code = NotFound desc = could not find container \"21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e\": container with ID starting with 21c6d9f80a7644a9f4fb75b17b01bf94ba80070e17d72fac22375b6c9f27458e not found: ID does not exist" Oct 30 05:46:59.194428 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.194387 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:59.208173 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.208147 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f97a49/simulator-single-main-f97a49-kserve-64fb65d4dd-tsnq4"] Oct 30 05:46:59.919942 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.919911 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:46:59.920447 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:46:59.920174 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" containerID="cri-o://8faec6a090f228a4bc0152642242634f75a8e18de3bcbf90e5c4d6e1cc26d4e0" gracePeriod=30 Oct 30 05:47:01.159809 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:01.159767 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979cc233-3759-4e11-8158-21ac6677ebd8" path="/var/lib/kubelet/pods/979cc233-3759-4e11-8158-21ac6677ebd8/volumes" Oct 30 05:47:05.168241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.168188 5058 generic.go:356] "Generic (PLEG): container finished" podID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerID="8faec6a090f228a4bc0152642242634f75a8e18de3bcbf90e5c4d6e1cc26d4e0" exitCode=0 Oct 30 05:47:05.168653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.168248 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerDied","Data":"8faec6a090f228a4bc0152642242634f75a8e18de3bcbf90e5c4d6e1cc26d4e0"} Oct 30 05:47:05.168653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.168318 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" event={"ID":"83a7beec-f194-4a62-80f8-89005d2e1d83","Type":"ContainerDied","Data":"de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2"} Oct 30 05:47:05.168653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.168349 5058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de11d24c8851fb0e93d33a543124fbdee527e3bacd756707b272f247caf7dde2" Oct 30 05:47:05.168653 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.168372 5058 scope.go:122] "RemoveContainer" containerID="2b605319ac24d037f4b32dd991009f95feda704106d7278790947f49dc7ee82b" Oct 30 05:47:05.182201 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.182181 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:47:05.299966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.299875 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.299966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.299940 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.299979 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300031 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300075 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300250 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300309 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300295 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:05.300420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300336 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300420 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300397 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\") pod \"83a7beec-f194-4a62-80f8-89005d2e1d83\" (UID: \"83a7beec-f194-4a62-80f8-89005d2e1d83\") " Oct 30 05:47:05.300662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300443 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:05.300662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300531 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:05.300662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300619 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:05.300831 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300771 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:05.301020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.300994 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a7beec-f194-4a62-80f8-89005d2e1d83-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.301020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.301021 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.301251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.301038 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.301251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.301053 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.301251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.301066 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.302327 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.302301 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:05.302445 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.302422 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "kube-api-access-jfcxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:05.302564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.302544 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:47:05.302740 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.302721 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token" pod "83a7beec-f194-4a62-80f8-89005d2e1d83" (UID: "83a7beec-f194-4a62-80f8-89005d2e1d83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:05.401668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.401634 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.401668 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.401666 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcxz\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-kube-api-access-jfcxz\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.401879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.401681 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.401879 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:05.401695 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a7beec-f194-4a62-80f8-89005d2e1d83-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:47:06.174008 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:06.173978 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb" Oct 30 05:47:06.274149 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:06.274115 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:47:06.279707 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:06.279680 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb"] Oct 30 05:47:07.159838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:47:07.159803 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" path="/var/lib/kubelet/pods/83a7beec-f194-4a62-80f8-89005d2e1d83/volumes" Oct 30 05:47:19.189908 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:47:19.189871 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:47:49.189453 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:47:49.189411 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:48:19.189641 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:48:19.189602 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:48:44.662611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:44.662559 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:44.663440 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:44.663388 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k" podUID="be496eac-0d82-460a-8bda-85585235e8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:44.894401 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:44.894349 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:44.894737 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:44.894705 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn" podUID="007cda5b-cdb8-4bb6-a49f-636045e9e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:45.043425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:45.043332 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:45.043704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:45.043671 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx" podUID="8536d142-4135-4460-9674-02c89d76a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:47.960498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:47.960466 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:48:47.968854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:48:47.968822 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:48:49.189364 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:48:49.189327 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:49:15.089940 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.089889 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:49:15.090719 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090690 5058 cpu_manager.go:414] "containerMap: removing container" podUID="979cc233-3759-4e11-8158-21ac6677ebd8" containerName="main" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090720 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979cc233-3759-4e11-8158-21ac6677ebd8" containerName="main" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090743 5058 cpu_manager.go:414] "containerMap: removing container" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090753 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090768 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerName="maas-api" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090776 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerName="maas-api" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090820 5058 cpu_manager.go:414] "containerMap: removing container" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.090828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090830 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.091263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090968 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.091263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.090989 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="979cc233-3759-4e11-8158-21ac6677ebd8" containerName="main" Oct 30 05:49:15.091263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.091002 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a7beec-f194-4a62-80f8-89005d2e1d83" containerName="istio-proxy" Oct 30 05:49:15.091263 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.091015 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a244ebdd-7ded-433f-a5a8-19d7a4fd9532" containerName="maas-api" Oct 30 05:49:15.094783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.094754 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.114334 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.114301 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dcbb35-openshift-default-dockercfg-vb6t5\"" Oct 30 05:49:15.129485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.129458 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:49:15.248575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248539 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.248575 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248580 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.248859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248645 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.248859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248676 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.248859 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248708 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.249050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248846 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.249050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248899 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.249050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248935 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxrl\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.249050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.248973 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.350757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350658 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.350757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350731 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350788 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350821 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350852 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxrl\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350884 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350935 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.350965 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351151 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.351007 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.351151 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.351713 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.351686 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.364819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.352033 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.364819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.352324 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.364819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.352336 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.364819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.354857 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.364819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.360910 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.373682 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.373652 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.373840 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.373754 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxrl\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\") pod \"e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.412779 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.412751 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:15.718712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.718686 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:49:15.719297 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:49:15.719268 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8852512_03a8_4dbc_9342_83ac8e487b27.slice/crio-e76df9f541a28a7718efdaaf845ab29815848a9a8c2d67398ffc90e5883c3cce WatchSource:0}: Error finding container e76df9f541a28a7718efdaaf845ab29815848a9a8c2d67398ffc90e5883c3cce: Status 404 returned error can't find the container with id e76df9f541a28a7718efdaaf845ab29815848a9a8c2d67398ffc90e5883c3cce Oct 30 05:49:15.721482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.721438 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:15.721590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.721530 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:15.721590 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.721572 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:15.891016 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.890977 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerStarted","Data":"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd"} Oct 30 05:49:15.891208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.891045 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerStarted","Data":"e76df9f541a28a7718efdaaf845ab29815848a9a8c2d67398ffc90e5883c3cce"} Oct 30 05:49:15.936132 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:15.936078 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podStartSLOduration=0.936063773 podStartE2EDuration="936.063773ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:15.935607597 +0000 UTC m=+3928.905102585" watchObservedRunningTime="2026-10-30 05:49:15.936063773 +0000 UTC m=+3928.905558761" Oct 30 05:49:16.413200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:16.413163 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:16.418468 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:16.418442 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:16.895989 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:16.895957 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:16.897032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:16.897010 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:19.189775 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:49:19.189731 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:49:19.936692 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:19.936659 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:49:19.940506 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:19.940487 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:19.974574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:19.974538 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:49:20.005498 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.005455 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dcbb35-serving-cert\"" Oct 30 05:49:20.100418 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.100350 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.100783 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.100758 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.100930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.100870 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gzp\" (UniqueName: \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.101005 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.100929 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.201777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.201694 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.201777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.201744 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gzp\" (UniqueName: \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.202352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.201777 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.202352 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.201900 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.202650 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.202624 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.204313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.204286 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.204422 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.204297 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.221867 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.221839 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gzp\" (UniqueName: \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\") pod \"maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.253628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.253601 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:20.581999 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.581970 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:49:20.582370 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:49:20.582346 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7214de_7d99_47fb_b61c_b7c827070d90.slice/crio-e9355858877f9568e662f01105dbeb676882a46e071f2f5225a7cae0226fe7c5 WatchSource:0}: Error finding container e9355858877f9568e662f01105dbeb676882a46e071f2f5225a7cae0226fe7c5: Status 404 returned error can't find the container with id e9355858877f9568e662f01105dbeb676882a46e071f2f5225a7cae0226fe7c5 Oct 30 05:49:20.583665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.583648 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:49:20.916476 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:20.916441 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" event={"ID":"4e7214de-7d99-47fb-b61c-b7c827070d90","Type":"ContainerStarted","Data":"e9355858877f9568e662f01105dbeb676882a46e071f2f5225a7cae0226fe7c5"} Oct 30 05:49:21.923210 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:21.923172 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" event={"ID":"4e7214de-7d99-47fb-b61c-b7c827070d90","Type":"ContainerStarted","Data":"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8"} Oct 30 05:49:21.923693 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:21.923255 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:21.966076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:21.966018 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" podStartSLOduration=2.550001501 podStartE2EDuration="2.966002276s" totalImagesPullingTime="416.000775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:21.964971339 +0000 UTC m=+3934.934466327" watchObservedRunningTime="2026-10-30 05:49:21.966002276 +0000 UTC m=+3934.935497264" Oct 30 05:49:27.950838 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:27.950793 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:49:32.719965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.718746 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": read tcp 10.132.0.2:51206->10.132.1.9:15021: read: connection reset by peer" start-of-body= Oct 30 05:49:32.719965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.718812 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": read tcp 10.132.0.2:51206->10.132.1.9:15021: read: connection reset by peer" Oct 30 05:49:32.992076 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.991995 5058 generic.go:356] "Generic (PLEG): container finished" podID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerID="0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd" exitCode=0 Oct 30 05:49:32.992251 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.992071 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerDied","Data":"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd"} Oct 30 05:49:32.992461 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.992446 5058 scope.go:122] "RemoveContainer" containerID="0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd" Oct 30 05:49:32.993574 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.993549 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:32.993632 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.993615 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:32.993669 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:32.993649 5058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:33.998821 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:33.998787 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerStarted","Data":"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602"} Oct 30 05:49:34.413147 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:34.413119 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:34.413452 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:34.413424 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:49:34.413512 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:34.413475 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 05:49:35.123600 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.123570 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:49:35.141930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.141897 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.155499 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.155465 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:49:35.167853 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.167823 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.168046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.167889 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.168046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.167975 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.168046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.167997 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.168046 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.168027 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gwhp\" (UniqueName: \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.168394 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.168082 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.172034 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.171998 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dcbb35\"/\"kube-root-ca.crt\"" Oct 30 05:49:35.185667 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.185638 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dcbb35\"/\"openshift-service-ca.crt\"" Oct 30 05:49:35.213628 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.213602 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dcbb35\"/\"default-dockercfg-xb7s7\"" Oct 30 05:49:35.213771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.213637 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dcbb35\"/\"simulator-single-main-dcbb35-kserve-self-signed-certs\"" Oct 30 05:49:35.269074 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269042 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269117 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269143 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269313 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269173 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gwhp\" (UniqueName: \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269345 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269392 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269644 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269601 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269712 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269660 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.269785 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.269764 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.271471 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.271450 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.271808 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.271790 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.293104 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.293077 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gwhp\" (UniqueName: \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\") pod \"simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.413501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.413466 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:49:35.413680 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.413521 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 05:49:35.481065 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.481027 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:35.797254 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:35.797207 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:49:35.797515 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:49:35.797496 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1881a831_6d14_42bb_8803_b820e9d4d8cb.slice/crio-9d604aef015a000754377dfcc8b014169290b7959fc25224a71b887f31d9409d WatchSource:0}: Error finding container 9d604aef015a000754377dfcc8b014169290b7959fc25224a71b887f31d9409d: Status 404 returned error can't find the container with id 9d604aef015a000754377dfcc8b014169290b7959fc25224a71b887f31d9409d Oct 30 05:49:36.010249 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.010194 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" event={"ID":"1881a831-6d14-42bb-8803-b820e9d4d8cb","Type":"ContainerStarted","Data":"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c"} Oct 30 05:49:36.010415 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.010263 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" event={"ID":"1881a831-6d14-42bb-8803-b820e9d4d8cb","Type":"ContainerStarted","Data":"9d604aef015a000754377dfcc8b014169290b7959fc25224a71b887f31d9409d"} Oct 30 05:49:36.066777 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.066661 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" podStartSLOduration=1.066646798 podStartE2EDuration="1.066646798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:36.066387858 +0000 UTC m=+3949.035882842" watchObservedRunningTime="2026-10-30 05:49:36.066646798 +0000 UTC m=+3949.036141824" Oct 30 05:49:36.413166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.413129 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:49:36.413618 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.413185 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 05:49:36.481391 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:36.481345 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:37.414400 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:37.414358 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:49:37.414881 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:37.414428 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 05:49:38.413837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:38.413795 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.414071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:38.413843 5058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 05:49:39.768362 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:39.768325 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:39.768931 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:39.768913 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:39.769872 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:39.769852 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:49:45.481488 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:45.481446 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:45.494115 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:45.494086 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:46.077703 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:49:46.077672 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:49:49.189597 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:49:49.189562 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:50:10.564357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:10.564320 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:50:16.442655 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:16.442622 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:50:16.443332 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:16.442850 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" podUID="4e7214de-7d99-47fb-b61c-b7c827070d90" containerName="maas-api" containerID="cri-o://5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8" gracePeriod=30 Oct 30 05:50:17.707709 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.707682 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:50:17.802963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.802881 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\") pod \"4e7214de-7d99-47fb-b61c-b7c827070d90\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " Oct 30 05:50:17.802963 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.802938 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\") pod \"4e7214de-7d99-47fb-b61c-b7c827070d90\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " Oct 30 05:50:17.803198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.803013 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\" (UniqueName: \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\") pod \"4e7214de-7d99-47fb-b61c-b7c827070d90\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " Oct 30 05:50:17.803198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.803051 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\") pod \"4e7214de-7d99-47fb-b61c-b7c827070d90\" (UID: \"4e7214de-7d99-47fb-b61c-b7c827070d90\") " Oct 30 05:50:17.803515 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.803488 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca" pod "4e7214de-7d99-47fb-b61c-b7c827070d90" (UID: "4e7214de-7d99-47fb-b61c-b7c827070d90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:17.804965 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.804939 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls" pod "4e7214de-7d99-47fb-b61c-b7c827070d90" (UID: "4e7214de-7d99-47fb-b61c-b7c827070d90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:17.805078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.805018 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs" pod "4e7214de-7d99-47fb-b61c-b7c827070d90" (UID: "4e7214de-7d99-47fb-b61c-b7c827070d90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:17.805078 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.805066 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp" pod "4e7214de-7d99-47fb-b61c-b7c827070d90" (UID: "4e7214de-7d99-47fb-b61c-b7c827070d90"). InnerVolumeSpecName "kube-api-access-k5gzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:17.904014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.903981 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:17.904014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.904010 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e7214de-7d99-47fb-b61c-b7c827070d90-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:17.904014 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.904020 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5gzp\" (UniqueName: \"kubernetes.io/projected/4e7214de-7d99-47fb-b61c-b7c827070d90-kube-api-access-k5gzp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:17.904311 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:17.904031 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7214de-7d99-47fb-b61c-b7c827070d90-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:18.232252 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.232194 5058 generic.go:356] "Generic (PLEG): container finished" podID="4e7214de-7d99-47fb-b61c-b7c827070d90" containerID="5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8" exitCode=0 Oct 30 05:50:18.232441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.232259 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" Oct 30 05:50:18.232441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.232255 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" event={"ID":"4e7214de-7d99-47fb-b61c-b7c827070d90","Type":"ContainerDied","Data":"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8"} Oct 30 05:50:18.232441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.232391 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc" event={"ID":"4e7214de-7d99-47fb-b61c-b7c827070d90","Type":"ContainerDied","Data":"e9355858877f9568e662f01105dbeb676882a46e071f2f5225a7cae0226fe7c5"} Oct 30 05:50:18.232441 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.232413 5058 scope.go:122] "RemoveContainer" containerID="5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8" Oct 30 05:50:18.242726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.242707 5058 scope.go:122] "RemoveContainer" containerID="5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8" Oct 30 05:50:18.243002 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:18.242986 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8\": container with ID starting with 5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8 not found: ID does not exist" containerID="5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8" Oct 30 05:50:18.243045 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.243012 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8"} err="failed to get container status \"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8\": rpc error: code = NotFound desc = could not find container \"5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8\": container with ID starting with 5560f0b26687a5d9a8e2955b3d41f999e3d964df9794114e6b049967feadf0e8 not found: ID does not exist" Oct 30 05:50:18.287697 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.287658 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:50:18.301331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:18.301298 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcbb35-79d887d64-7p2xc"] Oct 30 05:50:19.160120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:19.160081 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7214de-7d99-47fb-b61c-b7c827070d90" path="/var/lib/kubelet/pods/4e7214de-7d99-47fb-b61c-b7c827070d90/volumes" Oct 30 05:50:19.189634 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:19.189598 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:50:24.426645 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:24.426608 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:50:24.427161 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:24.426861 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" containerID="cri-o://edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602" gracePeriod=30 Oct 30 05:50:24.771807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:24.771730 5058 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:50:24.771807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:24.771786 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:50:25.585662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.585593 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" podUID="1881a831-6d14-42bb-8803-b820e9d4d8cb" containerName="main" containerID="cri-o://bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c" gracePeriod=45 Oct 30 05:50:25.845858 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.845800 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:50:25.886408 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886371 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\" (UniqueName: \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886471 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886533 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886580 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886616 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886725 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886698 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\") pod \"1881a831-6d14-42bb-8803-b820e9d4d8cb\" (UID: \"1881a831-6d14-42bb-8803-b820e9d4d8cb\") " Oct 30 05:50:25.886908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886783 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:25.886908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886854 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:25.886908 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.886896 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:25.887462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.887037 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.887462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.887053 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.887462 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.887067 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.889439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.889394 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:25.889439 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.889408 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "kube-api-access-9gwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:25.889871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.889850 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs" pod "1881a831-6d14-42bb-8803-b820e9d4d8cb" (UID: "1881a831-6d14-42bb-8803-b820e9d4d8cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:25.988179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.988149 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1881a831-6d14-42bb-8803-b820e9d4d8cb-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.988179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.988175 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gwhp\" (UniqueName: \"kubernetes.io/projected/1881a831-6d14-42bb-8803-b820e9d4d8cb-kube-api-access-9gwhp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.988179 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:25.988187 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1881a831-6d14-42bb-8803-b820e9d4d8cb-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:26.273775 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.273738 5058 generic.go:356] "Generic (PLEG): container finished" podID="1881a831-6d14-42bb-8803-b820e9d4d8cb" containerID="bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c" exitCode=0 Oct 30 05:50:26.273981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.273793 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" Oct 30 05:50:26.273981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.273826 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" event={"ID":"1881a831-6d14-42bb-8803-b820e9d4d8cb","Type":"ContainerDied","Data":"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c"} Oct 30 05:50:26.273981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.273870 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6" event={"ID":"1881a831-6d14-42bb-8803-b820e9d4d8cb","Type":"ContainerDied","Data":"9d604aef015a000754377dfcc8b014169290b7959fc25224a71b887f31d9409d"} Oct 30 05:50:26.273981 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.273887 5058 scope.go:122] "RemoveContainer" containerID="bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c" Oct 30 05:50:26.283116 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.283098 5058 scope.go:122] "RemoveContainer" containerID="bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c" Oct 30 05:50:26.283501 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:26.283478 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c\": container with ID starting with bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c not found: ID does not exist" containerID="bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c" Oct 30 05:50:26.283562 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.283512 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c"} err="failed to get container status \"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c\": rpc error: code = NotFound desc = could not find container \"bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c\": container with ID starting with bab7452985203fd64dc89fbcba09dace6d407f99fff5d69bbdbbf79d7262094c not found: ID does not exist" Oct 30 05:50:26.328926 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.328897 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:50:26.330602 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:26.330582 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dcbb35/simulator-single-main-dcbb35-kserve-6cfb6456c9-kt6r6"] Oct 30 05:50:27.159659 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:27.159618 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1881a831-6d14-42bb-8803-b820e9d4d8cb" path="/var/lib/kubelet/pods/1881a831-6d14-42bb-8803-b820e9d4d8cb/volumes" Oct 30 05:50:28.979044 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979009 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:50:28.979485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979470 5058 cpu_manager.go:414] "containerMap: removing container" podUID="4e7214de-7d99-47fb-b61c-b7c827070d90" containerName="maas-api" Oct 30 05:50:28.979533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979485 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7214de-7d99-47fb-b61c-b7c827070d90" containerName="maas-api" Oct 30 05:50:28.979533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979501 5058 cpu_manager.go:414] "containerMap: removing container" podUID="1881a831-6d14-42bb-8803-b820e9d4d8cb" containerName="main" Oct 30 05:50:28.979533 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979507 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1881a831-6d14-42bb-8803-b820e9d4d8cb" containerName="main" Oct 30 05:50:28.979630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979579 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="1881a831-6d14-42bb-8803-b820e9d4d8cb" containerName="main" Oct 30 05:50:28.979630 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.979594 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7214de-7d99-47fb-b61c-b7c827070d90" containerName="maas-api" Oct 30 05:50:28.984072 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:28.984046 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.015495 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.015462 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6bc7baf9-kserve-self-signed-certs\"" Oct 30 05:50:29.017489 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.017461 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:50:29.118906 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.118871 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.119086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.118916 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.119086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.119043 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.119086 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.119076 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.119200 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.119118 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59p6\" (UniqueName: \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.119277 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.119198 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220766 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220722 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220776 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220846 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220870 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220895 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59p6\" (UniqueName: \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.220946 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.220921 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.221261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.221234 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.221261 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.221249 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.221409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.221382 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.223143 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.223112 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.223412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.223396 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.242665 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.242599 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59p6\" (UniqueName: \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\") pod \"e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.306930 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.306893 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:29.608064 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.607704 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:50:29.615084 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:50:29.615057 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c378a9b_5975_40d8_a98d_7c81a03fe7dd.slice/crio-eef126210c457d7cba8b047a913dc48de14005c0de936dd8b1f466fd6cfec2ce WatchSource:0}: Error finding container eef126210c457d7cba8b047a913dc48de14005c0de936dd8b1f466fd6cfec2ce: Status 404 returned error can't find the container with id eef126210c457d7cba8b047a913dc48de14005c0de936dd8b1f466fd6cfec2ce Oct 30 05:50:29.685629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.685606 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:50:29.825695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825601 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.825695 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825657 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.825941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825752 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.825941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825825 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.825941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825877 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.825941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825897 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:29.825941 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825930 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.826324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825964 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.826324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.825999 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.826324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826007 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:29.826324 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826049 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\") pod \"c8852512-03a8-4dbc-9342-83ac8e487b27\" (UID: \"c8852512-03a8-4dbc-9342-83ac8e487b27\") " Oct 30 05:50:29.826521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826325 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:29.826521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826348 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:29.826521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826498 5058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-credential-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.826521 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826517 5058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.826706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826532 5058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-workload-socket\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.826706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826547 5058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8852512-03a8-4dbc-9342-83ac8e487b27-istiod-ca-cert\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.826706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.826528 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:29.827954 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.827933 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:29.828098 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.828082 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "kube-api-access-cpxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:29.828395 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.828377 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:29.828467 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.828391 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo" pod "c8852512-03a8-4dbc-9342-83ac8e487b27" (UID: "c8852512-03a8-4dbc-9342-83ac8e487b27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:50:29.927514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.927477 5058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-envoy\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.927514 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.927511 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxrl\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-kube-api-access-cpxrl\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.927718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.927525 5058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-podinfo\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.927718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.927536 5058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-data\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:29.927718 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:29.927547 5058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8852512-03a8-4dbc-9342-83ac8e487b27-istio-token\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:50:30.297953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.297910 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" event={"ID":"7c378a9b-5975-40d8-a98d-7c81a03fe7dd","Type":"ContainerStarted","Data":"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a"} Oct 30 05:50:30.298442 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.297964 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" event={"ID":"7c378a9b-5975-40d8-a98d-7c81a03fe7dd","Type":"ContainerStarted","Data":"eef126210c457d7cba8b047a913dc48de14005c0de936dd8b1f466fd6cfec2ce"} Oct 30 05:50:30.299769 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.299734 5058 generic.go:356] "Generic (PLEG): container finished" podID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerID="edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602" exitCode=0 Oct 30 05:50:30.299921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.299782 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" Oct 30 05:50:30.299921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.299801 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerDied","Data":"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602"} Oct 30 05:50:30.299921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.299849 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2" event={"ID":"c8852512-03a8-4dbc-9342-83ac8e487b27","Type":"ContainerDied","Data":"e76df9f541a28a7718efdaaf845ab29815848a9a8c2d67398ffc90e5883c3cce"} Oct 30 05:50:30.299921 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.299878 5058 scope.go:122] "RemoveContainer" containerID="edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602" Oct 30 05:50:30.307297 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.307268 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:30.309558 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.309538 5058 scope.go:122] "RemoveContainer" containerID="0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd" Oct 30 05:50:30.317837 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.317815 5058 scope.go:122] "RemoveContainer" containerID="edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602" Oct 30 05:50:30.318124 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:30.318102 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602\": container with ID starting with edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602 not found: ID does not exist" containerID="edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602" Oct 30 05:50:30.318187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.318132 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602"} err="failed to get container status \"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602\": rpc error: code = NotFound desc = could not find container \"edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602\": container with ID starting with edd49caa850be1a0cd27c48c9dc6948092321180cd918a8b0d38a49d7d4da602 not found: ID does not exist" Oct 30 05:50:30.318187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.318147 5058 scope.go:122] "RemoveContainer" containerID="0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd" Oct 30 05:50:30.318414 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:30.318393 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd\": container with ID starting with 0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd not found: ID does not exist" containerID="0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd" Oct 30 05:50:30.318484 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.318422 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd"} err="failed to get container status \"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd\": rpc error: code = NotFound desc = could not find container \"0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd\": container with ID starting with 0cf1cf3148aa6a236e59b46a43561ef79a5fcd5982918c20dde4bb7c6e723bbd not found: ID does not exist" Oct 30 05:50:30.341153 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.341109 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" podStartSLOduration=2.3410976740000002 podStartE2EDuration="2.341097674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:30.340810186 +0000 UTC m=+4003.310305172" watchObservedRunningTime="2026-10-30 05:50:30.341097674 +0000 UTC m=+4003.310592698" Oct 30 05:50:30.381183 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.381149 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:50:30.384358 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:30.384333 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2"] Oct 30 05:50:31.160208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:31.160162 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" path="/var/lib/kubelet/pods/c8852512-03a8-4dbc-9342-83ac8e487b27/volumes" Oct 30 05:50:39.307287 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:39.307250 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:39.319771 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:39.319729 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:39.414092 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:39.414057 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:50:45.792380 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.792343 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:50:45.792916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.792896 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:50:45.793004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.792919 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:50:45.793068 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.793046 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:50:45.793107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.793066 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:50:45.797898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.797876 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.814596 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.814562 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:50:45.826607 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.826575 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6bc7baf9-kserve-self-signed-certs\"" Oct 30 05:50:45.912527 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912493 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.912706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912546 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.912706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912625 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.912706 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912669 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.912916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912815 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:45.912916 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:45.912871 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59sm\" (UniqueName: \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014128 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014088 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014142 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014163 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014186 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014263 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014328 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014304 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59sm\" (UniqueName: \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014591 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014625 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014609 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.014800 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.014708 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.016559 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.016536 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.017002 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.016979 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.036641 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.036614 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59sm\" (UniqueName: \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\") pod \"e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.110755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.110663 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:46.404370 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.404340 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:50:46.405359 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:50:46.405334 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda624a71f_59fa_4073_9e5a_958040a634f2.slice/crio-a31576b84fd7271e4b8fec47a91d5feeb3a7a27492fe63ef4c45fc8f10b018fd WatchSource:0}: Error finding container a31576b84fd7271e4b8fec47a91d5feeb3a7a27492fe63ef4c45fc8f10b018fd: Status 404 returned error can't find the container with id a31576b84fd7271e4b8fec47a91d5feeb3a7a27492fe63ef4c45fc8f10b018fd Oct 30 05:50:46.444764 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:46.444734 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" event={"ID":"a624a71f-59fa-4073-9e5a-958040a634f2","Type":"ContainerStarted","Data":"a31576b84fd7271e4b8fec47a91d5feeb3a7a27492fe63ef4c45fc8f10b018fd"} Oct 30 05:50:47.450966 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:47.450916 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" event={"ID":"a624a71f-59fa-4073-9e5a-958040a634f2","Type":"ContainerStarted","Data":"2262debf23e81d5a352251b09ef1b3098bf8567d57c27c74fb3a4775220b505b"} Oct 30 05:50:47.497410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:47.497360 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" podStartSLOduration=2.497348228 podStartE2EDuration="2.497348228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:47.496191213 +0000 UTC m=+4020.465686207" watchObservedRunningTime="2026-10-30 05:50:47.497348228 +0000 UTC m=+4020.466843227" Oct 30 05:50:48.111281 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:48.111243 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:49.190169 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:50:49.190126 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:50:52.345376 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:52.345332 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:52.346020 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:52.345982 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:56.111058 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:56.111012 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:56.124375 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:56.124346 5058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:50:56.537990 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:50:56.537961 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:51:07.339374 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:07.339328 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:07.340025 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:07.339992 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:16.109531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:16.109491 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:51:17.641232 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:17.641190 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:51:19.205245 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:51:19.205189 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:51:31.130939 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.130864 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" podUID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" containerName="main" containerID="cri-o://661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a" gracePeriod=45 Oct 30 05:51:31.398004 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.397977 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:51:31.439531 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439496 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439550 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\" (UniqueName: \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439629 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439681 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439720 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439778 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\") pod \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\" (UID: \"7c378a9b-5975-40d8-a98d-7c81a03fe7dd\") " Oct 30 05:51:31.439894 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.439883 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:31.440266 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.440081 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:31.440359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.440322 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:31.441425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.440483 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.441425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.440513 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.442727 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.442704 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:31.442819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.442737 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:31.442819 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.442754 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6" pod "7c378a9b-5975-40d8-a98d-7c81a03fe7dd" (UID: "7c378a9b-5975-40d8-a98d-7c81a03fe7dd"). InnerVolumeSpecName "kube-api-access-d59p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:31.541611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.541575 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d59p6\" (UniqueName: \"kubernetes.io/projected/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-kube-api-access-d59p6\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.541611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.541605 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.541611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.541616 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.541611 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.541624 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c378a9b-5975-40d8-a98d-7c81a03fe7dd-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.856807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.856712 5058 generic.go:356] "Generic (PLEG): container finished" podID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" containerID="661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a" exitCode=0 Oct 30 05:51:31.856807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.856761 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" Oct 30 05:51:31.856807 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.856791 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" event={"ID":"7c378a9b-5975-40d8-a98d-7c81a03fe7dd","Type":"ContainerDied","Data":"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a"} Oct 30 05:51:31.857121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.856830 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt" event={"ID":"7c378a9b-5975-40d8-a98d-7c81a03fe7dd","Type":"ContainerDied","Data":"eef126210c457d7cba8b047a913dc48de14005c0de936dd8b1f466fd6cfec2ce"} Oct 30 05:51:31.857121 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.856847 5058 scope.go:122] "RemoveContainer" containerID="661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a" Oct 30 05:51:31.866253 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.866236 5058 scope.go:122] "RemoveContainer" containerID="661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a" Oct 30 05:51:31.866530 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:51:31.866509 5058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a\": container with ID starting with 661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a not found: ID does not exist" containerID="661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a" Oct 30 05:51:31.866576 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.866540 5058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a"} err="failed to get container status \"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a\": rpc error: code = NotFound desc = could not find container \"661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a\": container with ID starting with 661f9e9af0d25782356c0d90300d19682a38d21935f2a97cfc62fc35b075852a not found: ID does not exist" Oct 30 05:51:31.913663 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.913629 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:51:31.927024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:31.926991 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6bc7baf9-kserve-85bd69cbb5-cs4vt"] Oct 30 05:51:32.661827 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:32.661760 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" podUID="a624a71f-59fa-4073-9e5a-958040a634f2" containerName="main" containerID="cri-o://2262debf23e81d5a352251b09ef1b3098bf8567d57c27c74fb3a4775220b505b" gracePeriod=45 Oct 30 05:51:32.864698 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:32.864662 5058 generic.go:356] "Generic (PLEG): container finished" podID="a624a71f-59fa-4073-9e5a-958040a634f2" containerID="2262debf23e81d5a352251b09ef1b3098bf8567d57c27c74fb3a4775220b505b" exitCode=0 Oct 30 05:51:32.864871 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:32.864726 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" event={"ID":"a624a71f-59fa-4073-9e5a-958040a634f2","Type":"ContainerDied","Data":"2262debf23e81d5a352251b09ef1b3098bf8567d57c27c74fb3a4775220b505b"} Oct 30 05:51:32.943738 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:32.943714 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:51:33.054620 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054585 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.054841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054635 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\" (UniqueName: \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.054841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054667 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.054841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054689 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.054841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054717 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.054841 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.054820 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\") pod \"a624a71f-59fa-4073-9e5a-958040a634f2\" (UID: \"a624a71f-59fa-4073-9e5a-958040a634f2\") " Oct 30 05:51:33.055120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055024 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.055120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055072 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.055120 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055034 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.055267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055187 5058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-home\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.055267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055201 5058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-model-cache\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.055267 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.055212 5058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-tmp-dir\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.056943 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.056917 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.057054 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.057033 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:33.057097 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.057043 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm" pod "a624a71f-59fa-4073-9e5a-958040a634f2" (UID: "a624a71f-59fa-4073-9e5a-958040a634f2"). InnerVolumeSpecName "kube-api-access-d59sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.156033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.156003 5058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a624a71f-59fa-4073-9e5a-958040a634f2-tls-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.156033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.156029 5058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a624a71f-59fa-4073-9e5a-958040a634f2-dshm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.156033 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.156039 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d59sm\" (UniqueName: \"kubernetes.io/projected/a624a71f-59fa-4073-9e5a-958040a634f2-kube-api-access-d59sm\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.160637 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.160608 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" path="/var/lib/kubelet/pods/7c378a9b-5975-40d8-a98d-7c81a03fe7dd/volumes" Oct 30 05:51:33.888670 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.888632 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" event={"ID":"a624a71f-59fa-4073-9e5a-958040a634f2","Type":"ContainerDied","Data":"a31576b84fd7271e4b8fec47a91d5feeb3a7a27492fe63ef4c45fc8f10b018fd"} Oct 30 05:51:33.893381 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.893337 5058 scope.go:122] "RemoveContainer" containerID="2262debf23e81d5a352251b09ef1b3098bf8567d57c27c74fb3a4775220b505b" Oct 30 05:51:33.893781 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.893758 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5" Oct 30 05:51:33.970465 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:33.970339 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:51:34.114482 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:34.114433 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6bc7baf9-kserve-58847c6cdf-rfgc5"] Oct 30 05:51:35.178938 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:35.178894 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a624a71f-59fa-4073-9e5a-958040a634f2" path="/var/lib/kubelet/pods/a624a71f-59fa-4073-9e5a-958040a634f2/volumes" Oct 30 05:51:49.190508 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:51:49.190470 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:51:58.008914 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.008818 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jcgg"] Oct 30 05:51:58.009483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009459 5058 cpu_manager.go:414] "containerMap: removing container" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:51:58.009483 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009481 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8852512-03a8-4dbc-9342-83ac8e487b27" containerName="istio-proxy" Oct 30 05:51:58.009704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009545 5058 cpu_manager.go:414] "containerMap: removing container" podUID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" containerName="main" Oct 30 05:51:58.009704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009555 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" containerName="main" Oct 30 05:51:58.009704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009603 5058 cpu_manager.go:414] "containerMap: removing container" podUID="a624a71f-59fa-4073-9e5a-958040a634f2" containerName="main" Oct 30 05:51:58.009704 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009613 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a624a71f-59fa-4073-9e5a-958040a634f2" containerName="main" Oct 30 05:51:58.009917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009720 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c378a9b-5975-40d8-a98d-7c81a03fe7dd" containerName="main" Oct 30 05:51:58.009917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.009740 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a624a71f-59fa-4073-9e5a-958040a634f2" containerName="main" Oct 30 05:51:58.013235 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.013194 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.046726 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.046693 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jcgg"] Oct 30 05:51:58.113274 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.113210 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx86\" (UniqueName: \"kubernetes.io/projected/3fca16d7-827a-430b-9525-e2296ae83acc-kube-api-access-cbx86\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.113486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.113287 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-maas-api-tls\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.113486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.113344 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fca16d7-827a-430b-9525-e2296ae83acc-openshift-service-ca\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.113486 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.113369 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-metrics-certs\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.214250 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.214175 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx86\" (UniqueName: \"kubernetes.io/projected/3fca16d7-827a-430b-9525-e2296ae83acc-kube-api-access-cbx86\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.214412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.214269 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-maas-api-tls\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.214412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.214317 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fca16d7-827a-430b-9525-e2296ae83acc-openshift-service-ca\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.214412 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.214341 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-metrics-certs\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.215107 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.215073 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fca16d7-827a-430b-9525-e2296ae83acc-openshift-service-ca\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.217101 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.217071 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-metrics-certs\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.217207 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.217109 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fca16d7-827a-430b-9525-e2296ae83acc-maas-api-tls\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.234844 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.234811 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx86\" (UniqueName: \"kubernetes.io/projected/3fca16d7-827a-430b-9525-e2296ae83acc-kube-api-access-cbx86\") pod \"maas-api-595cc648c-4jcgg\" (UID: \"3fca16d7-827a-430b-9525-e2296ae83acc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.330609 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.330521 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:58.627163 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:58.627129 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jcgg"] Oct 30 05:51:58.628806 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:51:58.628778 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fca16d7_827a_430b_9525_e2296ae83acc.slice/crio-db6a5c11477c7353c63c5d4987c4b6c5cfbc062a18eb261d67bb283ee173d96d WatchSource:0}: Error finding container db6a5c11477c7353c63c5d4987c4b6c5cfbc062a18eb261d67bb283ee173d96d: Status 404 returned error can't find the container with id db6a5c11477c7353c63c5d4987c4b6c5cfbc062a18eb261d67bb283ee173d96d Oct 30 05:51:59.053245 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:59.053182 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" event={"ID":"3fca16d7-827a-430b-9525-e2296ae83acc","Type":"ContainerStarted","Data":"7a95122bcadf6598505c39791a73d27cbee6fee6de3c1629a3aa5a20ad584e5d"} Oct 30 05:51:59.053755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:59.053253 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" event={"ID":"3fca16d7-827a-430b-9525-e2296ae83acc","Type":"ContainerStarted","Data":"db6a5c11477c7353c63c5d4987c4b6c5cfbc062a18eb261d67bb283ee173d96d"} Oct 30 05:51:59.053755 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:59.053382 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:51:59.096847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:51:59.096790 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" podStartSLOduration=1.817559559 podStartE2EDuration="2.096774282s" totalImagesPullingTime="279.214723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:59.095464658 +0000 UTC m=+4092.064959647" watchObservedRunningTime="2026-10-30 05:51:59.096774282 +0000 UTC m=+4092.066269271" Oct 30 05:52:05.063855 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:05.063824 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jcgg" Oct 30 05:52:12.948178 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:12.948139 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 05:52:12.948642 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:12.948384 5058 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" podUID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" containerName="maas-api" containerID="cri-o://561ad18bc76be8101862894761f11582a75f7093c3df6bf9bef9bade84ef3c1e" gracePeriod=30 Oct 30 05:52:14.206079 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.206044 5058 generic.go:356] "Generic (PLEG): container finished" podID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" containerID="561ad18bc76be8101862894761f11582a75f7093c3df6bf9bef9bade84ef3c1e" exitCode=0 Oct 30 05:52:14.206491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.206102 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" event={"ID":"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb","Type":"ContainerDied","Data":"561ad18bc76be8101862894761f11582a75f7093c3df6bf9bef9bade84ef3c1e"} Oct 30 05:52:14.206491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.206150 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" event={"ID":"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb","Type":"ContainerDied","Data":"f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2"} Oct 30 05:52:14.206491 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.206172 5058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09f6439f288c2b9c11de87b27908c6b35910ed52cfc5838b61d4c7d0a2e5ca2" Oct 30 05:52:14.208964 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.208945 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 05:52:14.311508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.311412 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\") pod \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " Oct 30 05:52:14.311508 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.311463 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\" (UniqueName: \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\") pod \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " Oct 30 05:52:14.311716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.311516 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\") pod \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " Oct 30 05:52:14.311716 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.311607 5058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\") pod \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\" (UID: \"9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb\") " Oct 30 05:52:14.311898 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.311875 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca" pod "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" (UID: "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:14.312050 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.312029 5058 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-openshift-service-ca\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:52:14.313854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.313818 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs" pod "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" (UID: "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:14.313854 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.313836 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls" pod "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" (UID: "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:14.314006 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.313870 5058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp" pod "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" (UID: "9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb"). InnerVolumeSpecName "kube-api-access-v56pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:14.413187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.413152 5058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56pp\" (UniqueName: \"kubernetes.io/projected/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-kube-api-access-v56pp\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:52:14.413187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.413184 5058 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-maas-api-tls\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:52:14.413187 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:14.413195 5058 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb-metrics-certs\") on node \"ip-10-0-139-228.ec2.internal\" DevicePath \"\"" Oct 30 05:52:15.211059 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:15.211025 5058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx" Oct 30 05:52:15.268280 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:15.268210 5058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 05:52:15.281770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:15.281725 5058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-rxlxx"] Oct 30 05:52:17.181241 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:17.181117 5058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" path="/var/lib/kubelet/pods/9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb/volumes" Oct 30 05:52:19.190623 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:52:19.190584 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:52:37.348378 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:37.348331 5058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:37.349166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:37.349127 5058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs" podUID="3f051c2f-f800-48a2-9a5d-d66a741cdff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:49.189832 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:52:49.189780 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:52:50.946166 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:50.946131 5058 scope.go:122] "RemoveContainer" containerID="561ad18bc76be8101862894761f11582a75f7093c3df6bf9bef9bade84ef3c1e" Oct 30 05:52:50.955279 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:52:50.955254 5058 scope.go:122] "RemoveContainer" containerID="8faec6a090f228a4bc0152642242634f75a8e18de3bcbf90e5c4d6e1cc26d4e0" Oct 30 05:53:12.147891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:12.147856 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:53:12.802182 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:12.802140 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-gm2nc_cc2dd475-d48d-43e5-a32b-721786ce16b1/manager/0.log" Oct 30 05:53:13.543816 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:13.543785 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-xqtxn_faac0405-0672-447b-be1c-661e91ace85a/copy-manifests/0.log" Oct 30 05:53:13.574369 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:13.574344 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-xqtxn_faac0405-0672-447b-be1c-661e91ace85a/manager/0.log" Oct 30 05:53:14.541911 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:14.541883 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4jcgg_3fca16d7-827a-430b-9525-e2296ae83acc/maas-api/0.log" Oct 30 05:53:16.619763 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:16.619734 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nl2nj_21ca24c3-36fe-40f9-90a2-ef416baa6167/postgres/0.log" Oct 30 05:53:18.431485 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:18.431449 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-vsxt8_7050e41e-64d0-44ad-bfc8-d1ca9456930b/manager/0.log" Oct 30 05:53:18.654601 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:18.654568 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-9s8jz_2c2154fe-808a-4775-b983-8dee7295a515/manager/0.log" Oct 30 05:53:19.190092 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:53:19.190054 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:53:19.475397 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:19.475317 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n4k9r_0f58bc18-62b8-46a7-b845-d91cb7d936e1/manager/0.log" Oct 30 05:53:20.804425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:20.804395 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-kgtw4_287a26ad-2736-4bd4-a588-f384ad7de793/kube-auth-proxy/0.log" Oct 30 05:53:20.999017 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:20.998985 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs_3f051c2f-f800-48a2-9a5d-d66a741cdff4/istio-proxy/0.log" Oct 30 05:53:21.613828 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:21.613794 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-68n2k_be496eac-0d82-460a-8bda-85585235e8ce/istio-proxy/0.log" Oct 30 05:53:21.996105 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:21.996075 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx_8536d142-4135-4460-9674-02c89d76a082/istio-proxy/0.log" Oct 30 05:53:22.582492 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:22.582455 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-wprbn_007cda5b-cdb8-4bb6-a49f-636045e9e4e0/istio-proxy/0.log" Oct 30 05:53:23.374604 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:23.374571 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74b99df667-5m6dw_48383d31-9302-4010-867c-0162421153e4/router/0.log" Oct 30 05:53:24.393315 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:24.393278 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh_2ab64951-a948-4e61-9a3c-58438e9dadd5/main/0.log" Oct 30 05:53:24.589024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:24.588985 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-snw9d_5409a084-5c7a-4ad4-87db-98dfe91eadc3/main/0.log" Oct 30 05:53:25.172662 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:25.172629 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455_b292f152-c018-4589-a01b-e7f49c505f47/main/0.log" Oct 30 05:53:36.607430 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:36.607401 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:53:37.267710 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:37.267676 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-gm2nc_cc2dd475-d48d-43e5-a32b-721786ce16b1/manager/0.log" Oct 30 05:53:37.948672 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:37.948641 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-xqtxn_faac0405-0672-447b-be1c-661e91ace85a/copy-manifests/0.log" Oct 30 05:53:37.980273 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:37.980244 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-xqtxn_faac0405-0672-447b-be1c-661e91ace85a/manager/0.log" Oct 30 05:53:38.712511 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:38.712481 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4jcgg_3fca16d7-827a-430b-9525-e2296ae83acc/maas-api/0.log" Oct 30 05:53:40.804917 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:40.804883 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nl2nj_21ca24c3-36fe-40f9-90a2-ef416baa6167/postgres/0.log" Oct 30 05:53:42.222385 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:42.222355 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-vsxt8_7050e41e-64d0-44ad-bfc8-d1ca9456930b/manager/0.log" Oct 30 05:53:42.445192 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:42.445153 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-9s8jz_2c2154fe-808a-4775-b983-8dee7295a515/manager/0.log" Oct 30 05:53:43.312953 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:43.312921 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n4k9r_0f58bc18-62b8-46a7-b845-d91cb7d936e1/manager/0.log" Oct 30 05:53:44.451331 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:44.451294 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-kgtw4_287a26ad-2736-4bd4-a588-f384ad7de793/kube-auth-proxy/0.log" Oct 30 05:53:44.639373 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:44.639341 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs_3f051c2f-f800-48a2-9a5d-d66a741cdff4/istio-proxy/0.log" Oct 30 05:53:45.243414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:45.243380 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-68n2k_be496eac-0d82-460a-8bda-85585235e8ce/istio-proxy/0.log" Oct 30 05:53:45.650748 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:45.650717 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx_8536d142-4135-4460-9674-02c89d76a082/istio-proxy/0.log" Oct 30 05:53:46.218757 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:46.218727 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-wprbn_007cda5b-cdb8-4bb6-a49f-636045e9e4e0/istio-proxy/0.log" Oct 30 05:53:46.990988 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:46.990959 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74b99df667-5m6dw_48383d31-9302-4010-867c-0162421153e4/router/0.log" Oct 30 05:53:47.550526 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:47.550495 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-z26jh_2ab64951-a948-4e61-9a3c-58438e9dadd5/main/0.log" Oct 30 05:53:47.746125 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:47.746090 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-snw9d_5409a084-5c7a-4ad4-87db-98dfe91eadc3/main/0.log" Oct 30 05:53:47.990900 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:47.990873 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:53:48.000945 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:48.000918 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:53:48.324042 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:53:48.323957 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8r8455_b292f152-c018-4589-a01b-e7f49c505f47/main/0.log" Oct 30 05:53:49.191294 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:53:49.191260 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:54:19.191004 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:54:19.190966 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:54:19.443891 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:54:19.443807 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:54:19.746610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:54:19.746524 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:54:49.190053 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:54:49.190012 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:54:59.994198 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:54:59.994106 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:55:00.285040 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:00.284952 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d77b88d59-t8d5z_8e35e726-c96c-4dae-8978-76edcaf2916f/manager/0.log" Oct 30 05:55:01.485347 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.485307 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzzpg/must-gather-cjmmn"] Oct 30 05:55:01.485770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.485754 5058 cpu_manager.go:414] "containerMap: removing container" podUID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" containerName="maas-api" Oct 30 05:55:01.485770 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.485766 5058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" containerName="maas-api" Oct 30 05:55:01.485847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.485838 5058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6f1fa4-4bd9-4a72-b4ee-a354e6e0f9bb" containerName="maas-api" Oct 30 05:55:01.489109 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.489091 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.518834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.518796 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"openshift-service-ca.crt\"" Oct 30 05:55:01.518834 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.518824 5058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"kube-root-ca.crt\"" Oct 30 05:55:01.519060 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.518842 5058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzzpg\"/\"default-dockercfg-jv58p\"" Oct 30 05:55:01.521056 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.521033 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/must-gather-cjmmn"] Oct 30 05:55:01.554410 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.554374 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn84m\" (UniqueName: \"kubernetes.io/projected/2fdeae30-b545-4dac-91b1-c20496cd7868-kube-api-access-hn84m\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.554598 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.554497 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fdeae30-b545-4dac-91b1-c20496cd7868-must-gather-output\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.655268 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.655212 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fdeae30-b545-4dac-91b1-c20496cd7868-must-gather-output\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.655472 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.655328 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn84m\" (UniqueName: \"kubernetes.io/projected/2fdeae30-b545-4dac-91b1-c20496cd7868-kube-api-access-hn84m\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.655564 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.655543 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fdeae30-b545-4dac-91b1-c20496cd7868-must-gather-output\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.676360 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.676317 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn84m\" (UniqueName: \"kubernetes.io/projected/2fdeae30-b545-4dac-91b1-c20496cd7868-kube-api-access-hn84m\") pod \"must-gather-cjmmn\" (UID: \"2fdeae30-b545-4dac-91b1-c20496cd7868\") " pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:01.823157 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:01.823060 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" Oct 30 05:55:02.119170 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:02.119143 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/must-gather-cjmmn"] Oct 30 05:55:02.120455 ip-10-0-139-228 kubenswrapper[5058]: W1030 05:55:02.120430 5058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fdeae30_b545_4dac_91b1_c20496cd7868.slice/crio-29cf4280fe1417984858e9d58db390e435c603b220fc24f5805012908e8c06a6 WatchSource:0}: Error finding container 29cf4280fe1417984858e9d58db390e435c603b220fc24f5805012908e8c06a6: Status 404 returned error can't find the container with id 29cf4280fe1417984858e9d58db390e435c603b220fc24f5805012908e8c06a6 Oct 30 05:55:02.122815 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:02.122795 5058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:55:02.205610 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:02.205573 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" event={"ID":"2fdeae30-b545-4dac-91b1-c20496cd7868","Type":"ContainerStarted","Data":"29cf4280fe1417984858e9d58db390e435c603b220fc24f5805012908e8c06a6"} Oct 30 05:55:03.215847 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:03.215809 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" event={"ID":"2fdeae30-b545-4dac-91b1-c20496cd7868","Type":"ContainerStarted","Data":"177c3ebdb38cb58f8463f4661869ae2c546804baeae0fecec9bd936dab7fd30c"} Oct 30 05:55:04.222741 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:04.222701 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" event={"ID":"2fdeae30-b545-4dac-91b1-c20496cd7868","Type":"ContainerStarted","Data":"dddaa0b55d4910e44ecaf162bdeabe380bad44856752ef7f965fbe67e8b0986e"} Oct 30 05:55:04.267937 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:04.266560 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzzpg/must-gather-cjmmn" podStartSLOduration=2.306912752 podStartE2EDuration="3.266538704s" totalImagesPullingTime="959.625952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:04.265420464 +0000 UTC m=+4277.234915454" watchObservedRunningTime="2026-10-30 05:55:04.266538704 +0000 UTC m=+4277.236033690" Oct 30 05:55:06.139024 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:06.138996 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7tbzb_bcf81c88-4fb2-4655-bd29-f2114fad2d3a/global-pull-secret-syncer/0.log" Oct 30 05:55:06.676761 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:06.676734 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4j2vn_5183f685-7bd0-41b0-a632-9d6e56bc3951/connection-checker/0.log" Oct 30 05:55:06.812581 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:06.812552 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9ff65_93773016-1fde-4819-905c-a0b774ab1b97/connection-checker/0.log" Oct 30 05:55:06.968414 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:06.968333 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cddr7_6e9c14b8-58c1-4d2b-bd17-a94c6b3bfd92/connection-checker/0.log" Oct 30 05:55:07.407421 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:07.405008 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nrrc2_e0743f86-5e12-4a1d-baf8-fb1973fe4034/konnectivity-agent/0.log" Oct 30 05:55:07.842189 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:07.842104 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-228.ec2.internal_cd885c44a562061a49f43990cf7668c0/haproxy/0.log" Oct 30 05:55:14.075243 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:14.075165 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-vsxt8_7050e41e-64d0-44ad-bfc8-d1ca9456930b/manager/0.log" Oct 30 05:55:14.200772 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:14.200565 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-9s8jz_2c2154fe-808a-4775-b983-8dee7295a515/manager/0.log" Oct 30 05:55:14.760174 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:14.760119 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n4k9r_0f58bc18-62b8-46a7-b845-d91cb7d936e1/manager/0.log" Oct 30 05:55:16.206357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.206318 5058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k"] Oct 30 05:55:16.213386 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.213355 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.227425 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.227390 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k"] Oct 30 05:55:16.252566 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.252528 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-sys\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.252887 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.252864 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-lib-modules\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.253102 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.253085 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-podres\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.253359 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.253343 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7sgc\" (UniqueName: \"kubernetes.io/projected/11823ecf-9d94-40bd-b037-9584b8594bfb-kube-api-access-j7sgc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.253563 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.253547 5058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-proc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356164 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-sys\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356208 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356208 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-lib-modules\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356265 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-podres\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356325 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-sys\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356343 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sgc\" (UniqueName: \"kubernetes.io/projected/11823ecf-9d94-40bd-b037-9584b8594bfb-kube-api-access-j7sgc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.356551 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356441 5058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-proc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.357011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356586 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-proc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.357011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356676 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-lib-modules\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.357011 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.356677 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11823ecf-9d94-40bd-b037-9584b8594bfb-podres\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.385142 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.385099 5058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sgc\" (UniqueName: \"kubernetes.io/projected/11823ecf-9d94-40bd-b037-9584b8594bfb-kube-api-access-j7sgc\") pod \"perf-node-gather-daemonset-rv59k\" (UID: \"11823ecf-9d94-40bd-b037-9584b8594bfb\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.539185 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.539085 5058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:16.862357 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:16.862324 5058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k"] Oct 30 05:55:17.077629 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:17.077593 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-l2bhc_1a1c0f74-26d2-4294-aafe-1d8b22ed2fbc/cluster-monitoring-operator/0.log" Oct 30 05:55:17.390237 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:17.390190 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" event={"ID":"11823ecf-9d94-40bd-b037-9584b8594bfb","Type":"ContainerStarted","Data":"354ec59768eb920d410acd2c7fc688b1f4c246925f4acdc132833bebfee19a4d"} Oct 30 05:55:17.390705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:17.390283 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:17.390705 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:17.390303 5058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" event={"ID":"11823ecf-9d94-40bd-b037-9584b8594bfb","Type":"ContainerStarted","Data":"547ab87513fafcd008fbd38355ea274a732f8ce9feeb9995452f38a4d4b64a4d"} Oct 30 05:55:17.432339 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:17.432284 5058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" podStartSLOduration=1.432269728 podStartE2EDuration="1.432269728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:17.430434983 +0000 UTC m=+4290.399929977" watchObservedRunningTime="2026-10-30 05:55:17.432269728 +0000 UTC m=+4290.401764760" Oct 30 05:55:18.137012 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:18.136975 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrmsh_fa8bdfcb-fb42-4367-abff-dfe545a0628b/node-exporter/0.log" Oct 30 05:55:18.258398 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:18.258366 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrmsh_fa8bdfcb-fb42-4367-abff-dfe545a0628b/kube-rbac-proxy/0.log" Oct 30 05:55:18.378184 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:18.378155 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrmsh_fa8bdfcb-fb42-4367-abff-dfe545a0628b/init-textfile/0.log" Oct 30 05:55:19.196063 ip-10-0-139-228 kubenswrapper[5058]: E1030 05:55:19.196022 5058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zh2bq" containerName="extractor" Oct 30 05:55:20.251501 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:20.251465 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9pkhh_38619267-fa1a-42a4-b54c-abf89f69ba9f/prometheus-operator-admission-webhook/0.log" Oct 30 05:55:22.781750 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:22.781718 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-84nvd_5dae5dcf-7d46-4eba-a2c8-1193789e3c28/networking-console-plugin/0.log" Oct 30 05:55:23.406746 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:23.406715 5058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-rv59k" Oct 30 05:55:23.841658 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:23.841574 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/2.log" Oct 30 05:55:23.845735 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:23.845713 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wf2dd_80515748-a212-4b00-bec6-9b459babaa08/console-operator/3.log" Oct 30 05:55:24.943708 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:24.943666 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-fpjq6_53f7eed5-61a2-417e-a55d-90dcb994ad2d/download-server/0.log" Oct 30 05:55:26.211049 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:26.211017 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jxfs5_e6975d49-682f-42d6-9ff2-21b90112b948/volume-data-source-validator/0.log" Oct 30 05:55:29.095349 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:29.095315 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq5jn_801eb2b7-bb95-4b2f-95d6-44880fdf062c/dns/0.log" Oct 30 05:55:29.216751 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:29.216713 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq5jn_801eb2b7-bb95-4b2f-95d6-44880fdf062c/kube-rbac-proxy/0.log" Oct 30 05:55:29.464022 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:29.463996 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9k5j_37948e1b-b416-4df5-9c56-7ff2e7996e48/dns-node-resolver/0.log" Oct 30 05:55:30.778690 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:30.778652 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-646f79545f-ggjr6_0acfde38-04ec-4b8f-b0a9-4a58af82f32d/registry/0.log" Oct 30 05:55:31.045032 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:31.044955 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lphjh_bbaf9947-5532-47ce-aea5-dcb456c37684/node-ca/0.log" Oct 30 05:55:31.676304 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:31.676272 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-kgtw4_287a26ad-2736-4bd4-a588-f384ad7de793/kube-auth-proxy/0.log" Oct 30 05:55:31.744039 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:31.744013 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs_3f051c2f-f800-48a2-9a5d-d66a741cdff4/istio-proxy/0.log" Oct 30 05:55:31.944409 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:31.944333 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-68n2k_be496eac-0d82-460a-8bda-85585235e8ce/istio-proxy/0.log" Oct 30 05:55:32.074382 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:32.074352 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx_8536d142-4135-4460-9674-02c89d76a082/istio-proxy/0.log" Oct 30 05:55:32.247071 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:32.246997 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-wprbn_007cda5b-cdb8-4bb6-a49f-636045e9e4e0/istio-proxy/0.log" Oct 30 05:55:32.506464 ip-10-0-139-228 kubenswrapper[5058]: I1030 05:55:32.506390 5058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74b99df667-5m6dw_48383d31-9302-4010-867c-0162421153e4/router/0.log"